Serious render quality problem

Rhoel · Post by **Rhoel** » Sat Feb 16, 2008 7:40 pm

Sorry Mike, but this is serious, really really F@#$ serious.

I had noticed something several times and blamed my eyesight/monitor etc. I last week I saw it again and it prompted me to do a test. The result was one of those OMG moments. I did the same test in Combustion and two edit programs, they were fine. AS renderer has a big render bug.

Digital theory is you can import a 100 x 100 image, save it out and the result will be 100 x 100. every pixel will line up with the original.

But not in Anime Studio. The final image is 100 x 100 but the 1:1 pixel positioning has changed.

Here is the example:

On the left is the original image, a one pixel line. If I import it into combustion, Premier, Vegas etc and output it, the 1 pixel line is identical. On th right is the same output after Anime Studio. The line is aliased.

The image is magnified to show the difference. The actual effect is to make the image appear "soft".

I have played with this for nearly three hours now, trying to figure out why. It looks like the BG image is starting at 1,1 instead of 0,0, and the resulting aliasing is the original image being stretched (or compressed) to compensate for the 1 pixel error. Suddenly, problems I saw on the 2K testing makes sense, why there was discrepancy between the vector and rastered lines.

I can test this more tomorrow, see if I can recreate the look manually in photoshop.

But whatever is causing this is a very big, major problem. It should never ever happen. It needs to be fixed and fast as it affects every single user using a background: Everyone's work is substandard, not as sharp as it should be.

Sorry to be the bearer of bad tidings. I'd be interested in both Mike's and other peoples comments.

Rhoel

Source material:
original 1024 x 576 image

as output 1024 x 576 image

---------------------------
edit
---------------------------
This was completed on the PC version, I have not tested this on Macs or Linux. If you have either versions please can you test. Because of the error type, I suspect its on all versions.

additional edit for typo.

heyvern · Post by **heyvern** » Sat Feb 16, 2008 9:24 pm

I sent a response to your PM, but I will share some of my thoughts here.

I agree with you 100%, this IS a bug and it is probably a bad one in certain specific situations. So far I have not encountered any problems.

Not to make light of your issue but I don't see this as a big problem as you do. Obviously it is a problem. If you have options to work around it till it is addressed that might be the best solution. Maybe sharpening would hide any unwanted effects.

Are the images in the above example the results you get with our 2k images? What I am asking is are those problems as obvious? Would they be noticeable to the average viewer? Yes the rendered quality isn't "100%" but are the results totally unacceptable?

I use AS for print work. I have output to print resolution poster size Photoshop files and never saw any problems... but of course I don't use image layers in those situations. At those sizes a blurring of a few pixels is hardly noticeable.

In conclusion I agree with you completely I just don't see it being as bad as you make out. It is bad... I just don't see it as the end of the world as we know it.

-vern

mkelley · Post by **mkelley** » Sat Feb 16, 2008 9:49 pm

With any aliasing pretty obviously the worst case scenerio is when you have absolutely parallel lines to the background -- however, this is seldom the case. Indeed, it's almost NEVER the case.

With any other line I'd think you'd WANT aliasing -- this would actually be a good thing. Because pixels aren't square anything you do along a line that isn't completely vertical or horizontal needs to be aliased to look right. In those cases I can't even imagine why you *wouldn't* want the line to be aliased.

Far more important than this, though, is it would make your objects blend better. For example (and you seem to know compositing, Rhoel, since you use Combustion) when I'm doing effects in After Effects I actually need to add in aliasing around the objects or the effects don't work -- they won't look like they sit properly in the backgrounds.

I haven't been doing 2D animation enough to understand if the aliasing in AS is something I'd rather add myself (that is, render the backgrounds separately and add in the right compositing in AE) but for now I've been doing all my stuff in AS, backgrounds and all, and this tiny amount of aliasing not only seems fine, but it might not be enough.

That's just my basic take on it -- obviously different strokes for different folks, and if it bothers you then perhaps it should be an option in the next version. For me it sounds just about right (or not enough, if you get my drift -- if anything I'd like the option to add more aliasing if/when needed).

heyvern · Post by **heyvern** » Sat Feb 16, 2008 10:17 pm

You've got the "right idea" about antialiasing Mkelly but miss the problem Rhoel has with this bug.

It isn't the antialiasing around or above an image layer that is the problem, it is the antialiasing or blurring of pixels within an image layer that is "1 to 1" in output size. Technically that image should not change in any way at all. Layers composited over it should antialias, but the image should be "untouched".

This "blurring" or "shifting" of pixels would not be effected by antialising of the edges of other layers.

That may be the problem, it could be that AS is antialiasing EVERYTHING when it should only be addressing each layer or just the EDGES of a layer or its transparency mask.

-vern

mkelley · Post by **mkelley** » Sat Feb 16, 2008 11:49 pm

Well, I still can't see why that would apply to anything other than horizontal or vertical lines.

I'm off right now but I know you know what I'm talking about -- pixels will "staircase" when they are diagonal (or any fraction thereof). This staircasing needs to be antialiased or the edges will not appear well -- they will, in fact, be fuzzy when an aa image is not.

This is true because pixels are rectangular in nature -- it can't be avoided -- and although you can draw with vectors you have to output pixels. I would not want a program that didn't do this, as edges would be ragged indeed.

Or maybe I'm missing the point again -- but I have to run. If I get a chance I'll come back tomorrow and try posting an example of exactly what I mean (but I can't believe you don't understand what I'm talking about).

heyvern · Post by **heyvern** » Sun Feb 17, 2008 12:31 am

You are right!

However, if you've created an image in say, Photoshop, and you've "antialiased" everything yourself or let's say you have vertical or horizontal lines you don't want antialiased... the image is perfect just the way you want it and you save as png to use in AS...

AS will "change" that image at render time even if it is not scaled or reduced.

Yes you do want to antialias angled lines etc... AS does that with vector layers... but an image layer at 100% should NOT be antialiased or changed AGAIN by AS.

That is the crux of the issue that Rhoel has pointed out. He wants AS to leave his images alone... if they aren't scaled... or to actually render them "accurately". If they are changed at 100% out put they are changed in a similar way when scaled or moved.

Like I said before, I agree that this is a bug, I just don't have a problem with it... yet.

-vern

Samb · Post by **Samb** » Sun Feb 17, 2008 12:33 am

well, an layer option for "don't anti-aliasing this layer" would be nice...
but on the other hand.. I don't know WHAT mike is exactly doing...
I mean com'on.. where is he?

last post he did, was on the 22nd november...
so.. anyone knows what's up with that?
where is mike? we need updates!

mkelley · Post by **mkelley** » Sun Feb 17, 2008 6:13 am

Vern,

Okay, okay -- I get it. Background images brought in.

Because I don't do this (everything I do in AS is vector) this wouldn't even occur to me to be a problem.

If I *were* going to use an actual bitmapped background I'd be compositing in another program anyway -- wouldn't expect AS to handle my bitmaps as well as I can handle them myself.

But I agree that that particular thing (leaving bitmapped images alone) should be fixed -- sorry, just didn't quite get it.

Rhoel · Post by **Rhoel** » Sun Feb 17, 2008 9:13 am

Thanks for the PM Vern, good points as you repeated here.

There have been a catalogue of problems associated with bitmaps and the rendere. None are resolved and I suspect this is all interlinked. Some raster images on a slow track jitter all over the place, as if the register fins have fallen out. The result is unbroadcastable.

I have Combustion and can us it for composites rather than use AS. But if I have a scene where I must use AS (Because of camera moves) the effect will be even more dramatic as you are cutting to a softer lower contrast image. I noted before Greykid use Combustion for composites - is this bug the reason why? I don't know.

But the fundamental problem is why is the renderer doing this. I have my suspicions or hunch which I will test today. If you import the test image and look at it in the work area, the right hand and bottom pixel row are cut off. It shouldn't be. The camera aperture should be image (width+1) x (Height+1). Currently what is being displayed is not. If this is the problem, then it's a relatively minor change to the renderer, not a rebuild or whatever. If its a floating point math issue, then there is a big problem.

But whatever the underlying causes, it IS essential to have it fixed.

Straight import/export images simply must not be degraded like this. The program should not be 'blurring', degrading bitmaps. The broadcasters quality control guys will discover it in HD because you can see it easily on a HD broadcast monitor: I have seen output on HD flat screens and put it down to deficiencies of the previous studio's non broadcast monitors. This test shows I really was seeing what I saw.

Anime Studio is now Smith Micro's baby and they need to paying attention. They are pushing the product to industry and charging $200 per licence. With that comes responsibility.

The product is broken and it needs fixing.

Rhoel · Post by **Rhoel** » Sun Feb 17, 2008 9:18 am

mkelley wrote:If I *were* going to use an actual bit mapped background I'd be compositing in another program anyway

So how would you handle a big camera pan? You would do everything as static layers, then do the camera move in the 3rd party program or would you try match moving?

Tough call.

Rhoel

Rhoel · Post by **Rhoel** » Sun Feb 17, 2008 9:47 am

heyvern wrote:Maybe sharpening would hide any unwanted effects.

Then that would affect the vector line adversely.

Are the images in the above example the results you get with our 2k images? What I am asking is are those problems as obvious? Would they be noticeable to the average viewer? Yes the rendered quality isn't "100%" but are the results totally unacceptable?

This example is from a 1024/576 standard broadcast frame and yes you can see it on a broadcast monitor.

I need to test this but I think this affects the smaller guys more if they are shooting for mobile - you have less pixels, therefore the error will be larger.

In conclusion I agree with you completely I just don't see it being as bad as you make out. It is bad... I just don't see it as the end of the world as we know it.

point taken but to me, i don't see the point of shooting on 35mm film then using a cheap bloomy lens. Nothing should compromise the quality unless you deliberately elect to degrade the quality.

There are issues which have been outstanding for at least two or more years - the picture jitter for example. By saying it doesn't matter seems not to concentrate minds. When Mike was just Mike. selling a product for $50, bug fixes which went unnoticed were excusable. But Smith Micro are pushing this to Industry and discounting on their $200 product starts at 11 units. AS is no longer a hobbyist/indie product - the big guys are pushing it to the animation industry and larger studios: Pricing reflects that. I am currently working on a pilot 2K project - I now have to consider every shot - I now have to re-budget every shot for Combustion: That is very expensive if I have to start motion tracking the pans. For me, The original pleasure I had knowing we would be working 2K progressive is now gone. The quality if we use AS for the render will not be as good.

Knowing I am releasing substandard work disappoints me greatly. I can't look the customer in the eye and tell him we did out best.

Because I know we haven't.

Rhoel

Manu · Post by **Manu** » Sun Feb 17, 2008 11:03 am

And the annoying thing is that the old trick of doubling up the render-size and reducing it in post doesn't seem to work either.

On the other hand, importing a background that's double the size in AS seems to reduce the blurring. I haven't done a very thorough test of this, but that's my impression so far.

mkelley · Post by **mkelley** » Sun Feb 17, 2008 1:41 pm

Rhoel wrote:
mkelley wrote:If I *were* going to use an actual bit mapped background I'd be compositing in another program anyway
So how would you handle a big camera pan? You would do everything as static layers, then do the camera move in the 3rd party program or would you try match moving?

I'm not all that impressed with camera moves (or layer moves, for that matter) in AS, so, yes, I'd do the camera move in AE. I can do a lot of things in 2D+ in AE that I can't in AS anyway, and I'd for sure be adding my effects in there (so there's no way I'd be using AS for final compositing anyway).

Unlike you I don't see a $200 program as needing to deliver exceptional features -- then again, our studio (my former one. I'm retired now) thought nothing of paying tens of thousands for software. Our principle renderer cost ten times what AS Pro does -- that's just for the renderer.

To me AS Pro is worth around $500 or so -- adding in more features could easily justify this, but even now that's the value I see. If it were being carried by any of the big boys that's what it would cost, and it's no wonder that Toon Boom Studio (among the very many different versions of TB) is priced around that.

Rhoel · Post by **Rhoel** » Sun Feb 17, 2008 3:24 pm

Just ran some more test to figure things out.

The smaller the camera size, the easier it is to see what is going on.

My hunch over the camera size is right. The work area 'camera' is 1 pixel smaller than the imported image of the same size. A camera set to 100 x 100 pixels only displays 99 x 99 pixels.

Whether this is all the problem I don't know, but it's interesting.

here is the AS camera set at 50 x 50 with a 50 x 50 imported bitmap.

The red 1 pixel line is cut off on the right and base. The rendered result is so:

My theory (and only Mike can answer this) is:

If the original camera is 49 x 49 pixels, and you are shifting everything 1/2 pixel to the right to create the 50 x 50, then you will get the this 'aliased' blurred look. Theoretical but it fits the result we are seeing.

If that is the case, making the camera width + 1 will fix the problem.

Mike or Smith Micro need to look at this and come back with an answer.

I'll now need to go play in Combustion with layering and see if I can figure out how to use AS layer render with Combustion pans and other camera moves.

Rhoel

addendum: I have sent an email to techsupport (at) smitnmicro.com drawing heir attention to the issue, requesting action.

Fingers crossed.

heyvern · Post by **heyvern** » Sun Feb 17, 2008 6:32 pm

I think I'm "on to something".

When looking at AS files in a text editor I've noticed that values set to "0" or appear to be integers in the program are actually floats with up to 6 decimal places in the file code itself.

In the file format for the project with your image the "width" appears to be "3.555556". Remember that AS uses "ratios" to determine the screen size NOT pixels.

This would definately account for the pixel offset if it is rounding. From my limited experience with programming rounding errors seem to be big problem... just a theory.

If I change the the float for the ration IN THE FILE FORMAT not the AS file I get very subtle changes to the render. I will play with it and see if there is a way to edit the file format to get an accurate size.

-vern