Tuesday, August 5, 2008

Giving Flash CS3's compiler a little breathing room

If you've ever worked on a decent sized (100,000+ line) project in Flash CS3, you may have run into situations where you can't compile with optimizations turned on: it either gives you bizarre errors or gives you a blank .swf. Sometimes you can't compile at all with or without optimization, or have to disable traces, and even then you may not succeed to build. It gives you cryptic feedback like "error generating bytecode" or sometimes no error at all. Frustrating.

I've done contract work on a few such projects, and it's a serious pain in the butt - nobody seems to have any answers save for "make your program smaller". Great.

There were plenty of rumors on how to fix it most of which involved doing magical dances with your fingers crossed (like rebooting and just hoping it works). None of them worked well or often, and finally today I set out to get to the bottom of it.

Since Actionscript 3 compiles down to Java bytecode and the default max heap size for the Java runtime environment is 128MB, I figured it was likely that the JVM was running out of memory and dying during the optimization phase. Armed with a similar hunch from a colleague (the Fu), I figured maybe giving Java a little more breathing room would fix the problem.

You can change the max heap size for the JVM with the command line option -Xmx, how do we get Flash to pass arguments to the JVM?

In the end, you can't. There's no way to get Flash to pass custom command line options to the java compiler.

It was just this kind of thing that the JAVA_TOOL_OPTIONS environment variable is there for. It watches over us, lovingly passing our guidance to the JVM when we can't do it directly.

When a JVM is being instantiated it checks the environment variable JAVA_TOOL_OPTIONS and treats it exactly as if it was passed at the command line. So now we can pass our beloved -Xmx1024M to the compiler and increase our max heap size from 128MB to a gig, and bam - we can compile with optimizations now.

To set the JAVA_TOOL_OPTIONS environment variable (in Windows), right click my computer->properties, open the "advanced" tab and click "environment variables". Hit "new" for either system or user - it doesn't really matter which you use unless you have multiple people using the computer. Enter JAVA_TOOL_OPTIONS for the variable name and -Xmx512M, -Xmx256M, -Xmx1024M, whatever. Oh, and don't forget the M :-)

This is how to set environment variables in OSX.

Happy compiling. We're not done yet though: there are a few gotchas.

  1. The maximum heap size you can get in Java is 2 gigs, but -Xmx2G bails with a failed allocation. The highest value you can give without the jvm bailing is 1640M. That's right: 1641 kills it. Why 1640? Because Sun hates us, that's why. No really I have no idea why. *edit* The Fu informed me that on the Vista machine on which he tested this, it wouldn't work with anything above 1024M. When I got 1640M to work it was on XP SP2 in a virtual machine. I'm curious what the limit is on OSX.
  2. If you enter any malformed, invalid, or otherwise "sketchy" options Java will fail to create its virtual machine, and Flash will freak out and tell you that you need to reinstall Flash. You can make everything better again by just deleting the JAVA_TOOL_OPTIONS environment variable.
  3. If the JVM keeps dying and making Flash sad, open up a command prompt, head to C:\Program Files\Adobe\Flash CS3\JVM\bin and type "java". Hopefully you'll get some helpful feedback about exactly why Java's dying.
  4. There is no number 4.
I hope this helps someone, because it sure would have saved me a lot of time if I'd known this before.