Group layer Depth Sort: Sort layers by Y-Position

[Lukas]

Not sure if this is possible, but I'd love to be able to set a group to 'Sort layers layers by Y-Position' instead of by depth.

So if a layer is above another layer in height, it would be rendered behind the other layer.
And if a layer is below another layer, it would be rendered on top of the other layer.

Sub-layers' positions should be ignored, I imagine the pivot point of a rig just always being between his feet.

Is this do-able with a layerscript on a group perhaps?

If anyone wants to give it a shot or point me in the right direction, I'd be very grateful.

(I've tried doing it by depth, but it gives too many problems... I wish Moho had an orthographic camera, that would also solve this issue)

[hayasidist]

instinct drives me away from a layerscript … interference with animated layer order, actual depth sort etc **might** cause problems.
I'd probably adopt a semi-automatic approach - have a (manually invoked) tool script that scans the timeline, does a "sort by y" and then keys an "animated layer order" when necessary.

e.g. (not actually tried this, so be aware that this might be buggy!)

on start up -
get the group layer;
set up for animated layer order and no depth sorting; (e.g. http://mohoscripting.com/methods/653)
clear the layer ordering channel (e.g. http://mohoscripting.com/methods/655)
create a table of child layers.


for all frames
> get all layer y values into the table
> sort that table by y -- order the layers so that the largest y is at the top of the layer stack
> has the layer order changed? if so
>> behindThis = first -- i.e. largest y
>> for all table entries except the first
>>> moveLayer = table entry
>>> moho:PlaceLayerBehindAnother(moveLayer, behindThis) -- http://mohoscripting.com/methods/67
>>> behindThis = moveLayer

[Lukas]

Thanks Hayasidist
I'm stubbornly trying a layerscript solution though...
Not sure if I'll stick to it, but it works for now. I think having to do it by hand will create messy Moho scenes.

It actually works pretty good:


LK_SortByYPos.lua

Code: Select all

-- **************************************************
-- Embed this layerscript to a group layer and it's child layers will be ordered by Y translation.
-- **************************************************
function LayerScript(moho)
	if (moho.layer:LayerType() == MOHO.LT_GROUP or moho.layer:LayerType() == MOHO.LT_BONE) then
		moho:LayerAsGroup(moho.layer):EnableLayerOrdering(true)
		local order, layer, yPos = {}
		local layerOrderChan = moho.layer:GetLayerOrdering()
		for i = moho.layer:CountLayers()-1, 0, -1 do
			layer = moho.layer:Layer(i)
			yPos = -layer.fTranslation.value.y -layer:Origin().y
			table.insert(order, {i, yPos})
		end
		table.sort(order,
			function(a,b)
				return (a[2]<b[2])
			end)
		local newOrder = ""
		for i,v in ipairs(order) do
			newOrder = newOrder .. moho.layer:Layer(v[1]):UUID() .. "|"
		end
		layerOrderChan:SetValue(moho.frame, newOrder)
		order = nil
	else
		print ("''LK_SortByYPos.lua'' should only be embedded into group or bone layers, ''" .. moho.layer:Name() .. "'' is not a group or bone layer.")
	end
end

[Maestral]

Actually, seeing it work does make it even more awesome (kudos Lukas) but I still can't figure out what was the problem with depth sort initially?
How comes this Y sort is a better solution than Z one? Care to add a line or two?

[Lukas]

Actually, seeing it work does make it even more awesome (kudos Lukas) but I still can't figure out what was the problem with depth sort initially?
How comes this Y sort is a better solution than Z one? Care to add a line or two?

Thanks
It's not *better* per se, but in this style it's a great solution. There's no proper perspective in this style, and using the Z axis would just add noise. For example, a character would become smaller because it's farther away (you could scale the group layer's Z axis to 0%, but that leads to new issues). I want to choose the scale of every character manually instead. I've got many shots with similar very flat layouts, so this way of doing things is hopefully going to save a lot of time, because the layer ordering is always up to date. You can easily add another vehicle to the scene and it would still all make sense.
This way of ordering would not work in many other situations.
Even with these flat layouts with a high horizon it might not always order things correctly (walking up a staircase in the foreground for example, it would put a character behind another one that should be behind it). But there's always exceptions, I'd just do it manually in those scenes.

[Maestral]

Many thanks for the explanation.
Most likely I've had not been in such situations but it's always good to know about the available workaround.

[Víctor Paredes]

Thanks Hayasidist
I'm stubbornly trying a layerscript solution though...
Not sure if I'll stick to it, but it works for now. I think having to do it by hand will create messy Moho scenes.

It actually works pretty good:

Wa! that looks amazing

[Greenlaw]

Hey, that's pretty cool!

When you first described what you wanted to do, I had trouble imagining how that would be useful. This clever example makes the usage very clear. Good job!

[Víctor Paredes]

I have been playing with the script for a while. Just playing. It's so much fun.
I'm not sure if it's useful, but do you thing it could be applied to scale too? For some projects, it would be nice to have the size changing too.
(I'm thinking it could take the scale of the top and bottom objects in the scene and calculate the size increase/decrease of the rest based on that).
But again, it just sounds fun to try. The script the way it is right now is extremely useful and I can already image using it in my projects

[jahnocli]

This is one of those things that you didn''t know you needed until you see it, then you can imagine all the things you could do with it! Thanks for sharing.

[Lukas]

Thanks guys.

Victor, I'm sure that auto-scale is possible, but in this case I'm sticking to doing it manually, because the design works better that way. Maybe someone wants to give it a try? I might try in the future, it's a good idea.

At first I thought you meant sorting by scale, which gave me an idea! Sorting by Z scale could make dynamic smartbone-controlled layer order a thing! So you could have a smartbone for the left arm that goes up/down the order, and another one for the right arm that does the same. And they would work together too, which solves the big layer-order-problem that I've been struggling with forever. Going to give it a shot today.

[Lukas]

Definitely onto something here!
This rig has 0 layer-order-keys and is completely 2D (even if you orbit the canvas, the Z axis is not used except for scale, which doesn't do anything for 2D planes facing the camera).
Smartbones now basically control the layer order trough Z scale keys, and the layerscript does the rest. This is a better and more flexible setup than anything I've tried in the past. Hopefully the layerscript doesn't make Moho crash when there's a lot of rigs in a scene.
I'm going to try and implement it in an actual rig and see if it holds up.
Here's the script if you want to give it a shot, I'm sure there's plenty of room for improvements: LK_SortByZScale.lua

[Greenlaw]

That's an interesting approach to animating 'wraps'. The rigs I create at work behave much like that but the setup process takes a lot of work and time. I'd like to try replicating a rig using your script and see if it simplifies or at least speeds up the process.

Thanks for making it available Lukas!

[Lukas]

Greenlaw, I'm curious about your current solution. Do you use layer-order keys controlled by smartbones? Or small z-translation values? Or some tricks with a lot of duplicate arms etc? Or some other magic trickery? I've tried so many ways in the past and always thought it was a dirty over-complex way of doing things and I'm dying for a proper solution.

I like this new approach, because it basically gives each layer a tweenable value, so smartbones can actually blend the values and it's not 1 solid inflexible layer-order key. (Unity works like this with sprites and it is the best system I've ever used) I prefer not having a lot of duplicate arms and stuff because it doesn't scale well and animators get confused and make a mess.

I think the trick here is finding the right values for each layer and its actions.

Right now I've got the layer's z scale values like this:
[body: '10'] and the arms at [arm1 (lowerarm/upperarm/hand): '3/5/6' and arm2: '13/15/16']

which can blend with a smartbone per arm to [arm1: '6/5/3' and arm2: '16/15/13'] (to change the direction, and also some values inbetween)
but also to [arm1: '13/15/16' and arm2 '3/5/6'] (to make them shift in front of and behind the body which always remains at 10)

Hope that makes sense.

I'm sure these values aren't the best ones, it would be nice to figure out a system that requires the least layers and smartbones. Here's the test file

[Greenlaw]

Greenlaw, I'm curious about your current solution. Do you use layer-order keys controlled by smartbones? Or small z-translation values? Or some tricks with a lot of duplicate arms etc? Or some other magic trickery?

By now, I've probably gone through just about every technique I can think of.

Originally, I used a reference with a mask and the mask position was controlled by an SBA. This generally worked but sometimes had issues in the elbow area. Later, I modified this using Stroke Exposure, which worked better but only for certain character designs (not so good for greatly exaggerated arm/leg shapes.)

Then I started duplicating limb layers and knocking out the upper and lower limbs where appropriate. With this setup, it was just hiding and revealing the segments based on the sorting position I wanted using a single SBA. Since the same continuous path was used for each of parts, the look was seamless and didn't have the problem in the joints like the masked version of the setup. The issue now is that I have to make four copies of each limb and, depending on the character design and complexity of the stacking requires, as many as eight copies. It's a lot to deal with on the rigging end but actually very nice to animate with. I do this mainly in rigs created for other artists to animate, artists who may not be very familiar enough with Moho rigging to modify the rigs as needed.

When I'm expected to animate the rigs myself, I might go with a simpler variation since I can hack and modify the rigs as I work through the shots. Personally, I'm careful not to build in too much 'automation' because I find that can backfire and restrict the animation possibilities. For me, it's faster to work this way too.

I generally avoid animating layer order in an SBA. The problem, I'm sure you know, is that this can lock the layer animation possibilities to only what the SBA does. If I'm planning to use Layer Order animation, I'd rather handle that manually. BTW, Wes' Layer Shortcuts makes this easier to do because the buttons can be set up to directly select specific groups and layers efficiently.

One trick I adopted from Victor is simplifying my limbs reveal order animation to a single SBA. For example, I might have one SBA to control all the possible arm reveal positions for both arms. I used to have separate SBA's for every limb. Multiple SBAs looks very organized and is very obvious to animate but, in practice, it's not necessary. Victor's approach is to simply add all the stacking/reveal possibilities inside a single SBA. This makes setting such things up easier and it also helps make the UI look less cluttered. It's also very easy to add more reveal possibilities when I need them.