Transforming a layer's bounding box

[avv]

With the MohoLayer:Bounds method, one can retrieve a BBox object with the fMin and fMax points of a layer. I need to find out how transform these points by the translation, scale and rotation of the layer in order to get the enclosing box around the transformed layer

But simply offsetting these points by the layer origin, scaling them, then applying the rotation and adding the translation won't do the job (In fact there seems to be an additional translation occurring in there somewhere )
After much deliberation, I found out how to properly calculate this box (see code). However it seems to only give correct results for image layers and not for group layers.

This is the code I wrote:

Code: Select all

local function rot(x, y, theta) 
	local cos, sin = math.cos, math.sin
	return x*cos(theta)+y*sin(theta), -x*sin(theta)+y*cos(theta)
end

local function getbox(l, f)
	local min, max = math.min, math.max

	local bbox = l:Bounds(f or 0)
	local o = l:Origin()
	local t = l.fTranslation:GetValue(0)
	local s = l.fScale:GetValue(f or 0)
	local r = l.fRotationZ:GetValue(f or 0)
	
	local ox, oy = o.x, o.y
	
	local minx, miny, maxx, maxy = bbox.fMin.x, bbox.fMin.y, bbox.fMax.x, bbox.fMax.y
	minx, miny, maxx, maxy = minx - ox, miny + oy, maxx - ox, maxy + oy
	minx, miny, maxx, maxy = minx * s.x, miny * s.y, maxx * s.x, maxy * s.y

	local ax, ay = 	rot(minx, miny, r)
	local bx, by = 	rot(minx, maxy, r)
	local cx, cy = 	rot(maxx, miny, r)
	local dx, dy = 	rot(maxx, maxy, r)
	
	local minx, miny, maxx, maxy = bbox.fMin.x, bbox.fMin.y, bbox.fMax.x, bbox.fMax.y
	minx, miny, maxx, maxy = minx + ox, miny - oy, maxx + ox, maxy - oy
	minx, miny, maxx, maxy = minx * s.x, miny * s.y, maxx * s.x, maxy * s.y
	
	local ex, ey = 	rot(minx, miny, r)
	local fx, fy = 	rot(minx, maxy, r)
	local gx, gy = 	rot(maxx, miny, r)
	local hx, hy = 	rot(maxx, maxy, r)

	return {
		minx= t.x + (min(min(min(ax, bx), cx), dx)) + ox, -- has to offset by ox and oy?
		miny= t.y + (min(min(min(ey, fy), gy), hy)) + oy,
		maxx= t.x + (max(max(max(ax, bx), cx), dx)) + ox,
		maxy= t.y + (max(max(max(ey, fy), gy), hy)) + oy,
	}
end

I need this to work for group layers as well. Could someone please enlighten me on how Moho transforms it's layers, and how they differ from image and group layers?
Example code is welcome.