Falcon Overview

Falcon is a new compiler for Flex that is under development. It will eventually replace the legacy compiler that has been part of the Flex SDK.

Current Status

Adobe donated the source code to the Falcon compiler as revision 1383121 on September 10, 2012. It currently lives at flex/falcon/trunk. A development branch at {flex/falcon/branches/develop}} will follow after the build scripts have been tested and debugged.

The ActionScript side of Falcon is near-shippable quality and was the basis for Flash Builder 4.7 Preview 1. The rest of Falcon (MXML, CSS, FXG, PROPERTIES) is pre-alpha quality and not ready for production use. Nevertheless, Falcon already supports enough of MXML to compile a correctly-running version of Checkinapp, the test app that is part of the SDK. (Exception: Checkinapp uses one Repeater tag, which is not yet implemented.) The main areas where Falcon needs more work are databinding, states, MXML-specific error reporting, and ASDoc generation.

Gordon Smith of Adobe, who has worked on Falcon from the beginning of the Falcon project, is on loan to Apache Flex for one day a week to improve Falcon's MXML support, with the goal of making it a suitable replacement for the legacy compiler. He will provide advice to Apache developers contributing to Falcon. Alex Harui of Adobe has become familiar with MXML code generation in Falcon and can offer advice in that area.

Everyone can contribute to Falcon's development by using it to compile their Flex applications and reporting any bugs you find in Jira. Reducing a bug to a simple test case will make it infinitely more likely to be fixed.

The following initial development efforts are planned:

• Run the source code for all Apache Flex SWCs through Falcon to see if spurious problems are reported.
• Update the source code as necessary, given the fact that Falcon is somewhat stricter than the legacy compiler. It should be possible to make both compilers happy.
• Get various test apps besides Checkinapp compiling, linking against Falcon-compiled SWCs, and running correctly.
• Get Mustella tests compiling with Falcon and passing.
• Fix bugs that the Apache Flex community prioritizes.

Falcon's Original Goals

Falcon had the following goals when the project was started. Compared with the legacy compiler,

• It should compile faster for both full and incremental compiles, especially for multi-project workspaces.
• It should require less memory, especially for multi-project workspaces.
• It should be useful as a code-intelligence engine and incremental compiler for an integrated development environment, and not just as a command-line compiler.
• It should have a design that is easy to understand and to evolve.
• It should generate better bytecode for improved runtime performance.

Source Code

The Falcon 'develop' branch is now available in Apache's GIT repository at https://git-wip-us.apache.org/repos/asf/flex-falcon.git. You can download it with any GIT client.

Binary Distributions

No binary distributions are available. Perhaps someone will volunteer to set up automated builds and a build download page?

Setup for Falcon

Falcon is written in Java 1.6. Its build scripts use Ant 1.7.1. Eclipse projects for development use Eclipse 4.2 (Juno).

You must obtain JFlex 1.4.3 as a prerequisite and set the environment variable JFLEX_JAR to point to JFlex.jar. You can obtain JFlex here.

Building Falcon

The top-level Ant script currently has the following targets:

The main, eclipse, or javadoc target will do a one-time download of five third party JARs:

• lib/antlr.jar
• lib/commons-cli.jar
• lib/commons-io.jar
• lib/guava.jar
• lib/lzma-sdk.jar

Occasionally one of these downloads hangs. Just interrupt Ant and try again. These JARs will remain in the lib directory until you do a wipe.

If you do ant -q main (where the -q switch means "quiet" and turns off a lot of noisy output), the output the first time should look like this:

$ ant -q
    [echo] JFLEX_JAR is d:/jflex-1.4.3/jflex/lib/JFlex.jar
    [echo] Obtaining lib/antlr.jar
    [echo] Obtaining lib/commons-cli.jar
    [echo] Obtaining lib/commons-io.jar
    [echo] Obtaining lib/guava.jar
    [echo] Obtaining lib/jburg.jar
    [echo] Obtaining lib/lzma.jar
    [echo] Building lib/lzma.jar
    [echo] Generating RawASTokenizer
    [echo] Generating RawASDocTokenizer
    [echo] Generating RawMXMLTokenizer
    [echo] Generating ASParser and ASTokenTypes
   [antlr] ANTLR Parser Generator   Version 2.7.7 (20060906)   1989-2005
    [echo] Generating MetadataParser and MetadataTokenTypes
   [antlr] ANTLR Parser Generator   Version 2.7.7 (20060906)   1989-2005
    [echo] Generating CSSLexer and CSSParser
    [echo] Generating CSSTree
    [echo] Generating CmcEmitter
    [echo] Generating CSSEmitter
    [echo] Compiling Java code
    [echo] Creating generated/dist/sdk/lib/compiler.jar
    [echo] Creating generated/dist/sdk/lib/falcon-asc.jar
    [echo] Creating generated/dist/sdk/lib/falcon-mxmlc.jar
    [echo] Creating generated/dist/sdk/lib/falcon-compc.jar
    [echo] Creating generated/dist/sdk/lib/falcon-optimizer.jar
    [echo] Creating generated/dist/sdk/lib/falcon-swfdump.jar
    [echo] Creating lib/aet.jar
    [echo] Creating generated/dist/sdk/lib/flexTasks.jar
    [echo] Copying Apache SDK
    [echo] Building support.swc
    [echo] compiler main completed on 09/14/2012 11:32:37 PM

BUILD SUCCESSFUL
Total time: 1 minute 16 seconds

From this you can see that the overall flow of the build is

• Download the necessary third-party JAR files.
• Generate Java-based lexer classes from JFlex .lex files, Java-based parser classes from ANTLR .g files, and Java-based BURM classes from JBurg .jbg files.
• Compile the Java code (both the code that's in GIT and the lexers/parsers/BURMs that are generated by the previous step).
• Create the JAR files that are the Falcon deliverables.
• Create a SWC file that needs to be used when compiling Flex with Falcon (because Falcon generates different code for CSS than the old compiler).

All the compiled Java classes that comprise Falcon are packaged as compiler.jar, which is generated at generated/dist/sdk/lib. Other JAR files such as falcon-mxmlc.jar and falcon-compc.jar are codeless launcher JARs which serve only to run a particular entry point within Falcon. Launchers such as the shell script mxmlc or the batch file mxmlc.bat simply run these launch JARs from the command line.

Because Falcon requires a number of other files (such as flex-config.xml, framework.swc, etc.) to compile anything, the Falcon build script actually creates a small SDK in which everything is arranged so that tools like Falcon's mxmlc work the way they would if they were in an actual Apache Flex SDK. This "Falcon SDK" is built at generated/dist/sdk.

Committing to Falcon

Each commit to the trunk will be expected to:

• Keep the Java warnings in the Eclipse project(s) at 0.
• Keep the 'ant javadoc' warnings at 0.
• Provide Javadoc for new non-private APIs.
• Not change the public API surface (to ease possible re-integration into Flash Builder).
• Pass whatever tests are part of ant tests, once we have some tests.

Using Falcon

On the Command Line

The bin folder inside generated/dist/sdk has the command-line launchers like mxmlc, so using Falcon is the same as you are used to. For example, if you have but this bin directory on your PATH and you have cd'd to you application directory, then you just do

mxmlc MyApp.mxml

The options to the Falcon version of mxmlc are basically the same as for the legacy compiler. Some options such as -keep are no longer supported. If you try use an unsupported options, a warning will let you know.

In an Ant Script

Like the legacy compiler, Falcon provides <mxmlc> and <compc> Ant tasks. They are implemented by the classes MXMLCTask and COMPCTask in the org.apache.flex.compiler.ant package. These classes are part of falcon.jar. To use these tasks in an Ant script, first do

<taskdef ???/>

The require that the Ant property FLEX_HOME be set to a Flex SDK directory so that config files and SWC files can be found.

Inside Eclipse

The relevant entry points for a Debug Configuration are the MXMLC and COMPC classes in the org.apache.flex.compiler.clients package.

Inside Flash Builder

There is no way to use Apache Falcon inside of Flash Builder, at least for now. Because Falcon is designed to be both a compiler and a code-intelligence engine, it has to be tightly integrated with Flash Builder; Flash Builder cannot load it from an SDK.

Testing Falcon

Falcon's existing test suites are not yet ready for donation, because they are very large and require more extensive review. They will be donated later.

As soon as possible after donation, a handful of tests will be developed to ensure that are still able to compile framework.swc and Checkinapp. These make poor unit tests but reasonable functional tests.

Javadoc

Falcon has reasonably extensive Javadoc, including a short overview and package-level documentation that provides a general orientation to various subsystems. The Javadoc is not currently available online. If we get an automated build for Falcon, we may start posting Javadoc.

To build the Javadoc, download the source code and do

ant javadoc

in the compiler directory within Falcon's trunk. The generated HTML is inside generated/javadoc within Falcon's trunk. Open index.html in any browser and start by clicking on the Description link at the top to see the overview.

History

Spring 2010

Falcon began as a project at Adobe in response to the Flex community's concerns about long compilation times with the legacy compiler. The project began with several months of discussions about improving the old compiler versus starting over with Falcon. The main reasons for starting over were:

*The compiler team at Adobe wanted a compiler that could do double-duty as the code-intelligence engine for Flash Builder, rather than being something completely separate that Flash Builder called just to create a SWF or SWC. This approach would avoid duplicate data structures and duplicate algorithms, and would enable live error highlighting based on accurate semantic analysis. The team ended up taking Flash Builder's code intelligence engine, improving it, and adding semantic analysis and code generation to it. The resulting Falcon compiler doesn't have dependencies on Flash Builder though... it's the other way around.

*The legacy compiler wasn't designed for compilation on multiple threads. The team wanted a compiler that was designed with concurrency in mind from the beginning, as a primary way to get better – and scalable – performance.

*There was an awkward relationship between the legacy asc (which essentially understands only how to compile a single .as file) and the legacy mxmlc (which layers on top multi-file compilation and support for .mxml, .css, .properties, and .fxg), based on their historical development by two separate teams at Adobe. The team wanted a single compiler, designed by a single team, that had a unified architecture, unified data structures, and multi-file / multi-language support from the bottom up.

Fall 2011

A strategic shift at Adobe resulted a refocus on ActionScript and a defocus on Flex and MXML. Flex was donated to Apache, with a promise to finish the ActionScript side of Falcon and donate it by the end of 2012. Non-ActionScript support was left intact, at pre-alpha quality.

Summer 2012

Falcon was prepared for donation to Apache. By this point it has involved approximately 15 person-years of development effort.

The Future

It's up to us now! However, before striking out in radical new directions we should bring Falcon to the point where it can replace the legacy compiler.

Architecture

Falcon is designed from the beginning to support compiling multiple targets in multiple projects in a workspace, as exist in an IDE like Flash Builder. By contrast, the legacy asc was designed to compile a single AS file, and then the legacy mxmlc was built by another team on top of that.

Falcon uses multiple threads to compile multiple files at the same time. It can even compile multiple method bodies in the same file in parallel. The more cores you have, the faster it goes, unlike the legacy compiler. The multiple threads are managed using Java Futures (java.util.concurrent.IFuture<V>). The maximum number of thread allowed to be used is a small multiple of the number of cores.

Critical data structures such as the symbol table (which stores information about which classes are known, what methods they have, etc.) are shared across the entire workspace, to minimize memory usage.

Critical data structures are maintained in memory to support both compilation and IDE code intelligence in an efficient and consistent way. For example: If you open an ActionScript file in Flash Builder 4.7, Falcon builds a syntax tree and symbol table for it to support intelligent editing. Compiling the file requires just one additional code generation step. By contrast, in Flash Builder 4.6, which uses the legacy compiler in the SDK, the IDE builds its own parse trees and symbol tables to support editing, and then when you compile the compiler in the SDK builds another set of parse trees and symbol tables. This is slow and a waste of memory.

Falcon was designed from the beginning for incremental compilation of a complete workspace, so it tracks dependencies between compilation units in various projects. If you edit a file, it knows which other files it should recompile.

It can compile an app against a library project without creating the SWC for that library on disk.

Falcon understands .as, .mxml, .fxg, .css, and .properties files.

The parse trees for AS consist of nodes from about 100 classes, such as LiteralNode, BinaryOperatorPlusNode, FunctionNode, and ClassNode. The parse trees for MXML consist of nodes from about 50 classes, such as MXMLDocumentNode, MXMLInstanceNode, MXMLScriptNode, etc. The ActionScript-y parts of MXML are represented by AS nodes that are children of MXML nodes.

The symbol table consists of objects representing the things your code defines, such as ClassDefinition, FunctionDefinition, and VariableDefinition. These are stored in scope objects which form a hierarchy reflecting the lexical blocks of the source code.

Falcon uses third-party grammars like JFlex, ANTLR, and JBurg to generate various lexers, parsers, and code generators. The lexers for AS and MXML are written in JFlex .lex files. The lexer for CSS, and the parsers for AS, MXML, and CSS, are written in ANTLR .g files. The code generator is written in JBurg .jbg files. JBurg is a Java implementation of a Bottom-Up Rewrite Generator (BURG) which generates Java code for a Bottom-Up-Rewrite Machine (BURM). The BURM uses patterns and rules to find subtrees within in an abstract syntax tree that it can "reduce" to an intermediate form, typically a sequence of bytecode instructions. The process continues until the entire tree is reduced to ABC, which goes into a DoABC tag of a SWF file.

The typical data flow for an AS file is

.as file --> tokens --> abstract syntax tree --> file scope
                                             --> byte code and compilation problems

and for an MXML file is

.mxml file --> tokens --> XML DOM --> file scope
                                  --> abstract syntax tree -> byte code and compilation problems

To maximize performance, MXML is compiled directly to ABC, not to ActionScript source code or an ActionScript parse tree.

Falcon tracks where in the source every token/node/definition came from, to support operations like go-to-definition in Flash Builder.

Falcon has the general concept of a "problem", where a problem can end up being considered an error, a warning, or ignored. Problems are discovered by the semantic checker, which is part of the same phase of compilation as code generation. (This minimizes the number of times that the syntax tree must be traversed.) Falcon provides live problem highlighting to Flash Builder. When you pause during editing a file, the file is compiled and the problems show up as red squiggly underlines. Falcon tries to continue after almost all problems, so that you don't necessarily have to fix every problem before you can get a SWF to run.

Falcon includes libraries for reading and writing SWFs and SWCs files, and a library for reading and writing ActionScript byte code (ABC), at the level of individual instructions.

Falcon includes transcoders for the various types of embedded assets, which read, for example, a PNG file and convert it to the form required inside a SWF. It does NOT support font trancoding, because this relies on proprietary Adobe technology which Adobe has not donated to Apache. Instead, you must use a separate fontswf tool to convert your font files to SWF files and then embed the fonts from them.

FAQs

Please add questions here and people who contribute to Falcon will be happy to try to answer them.

Q: What is a code intelligence engine?

A: A library that supports smart editing in an IDE by offering code hinting, code completion, go-to-definition, find-references, refactoring, live error highlighting, etc. These things are useful as you write a program, even if you don't compile and run it.

Q: What is a lexer?

A: A lexer turns a source code file into a linear sequence of tokens. For example,

trace(i + 1);

would be lexed into seven token objects, represented as

TOKEN_IDENTIFIER "trace"
TOKEN_LEFT_PAREN "("
TOKEN_IDENTIFIER "i"
TOKEN_OPERATOR_PLUS "+"
TOKEN_NUMERIC_LITERAL "1"
TOKEN_RIGHT_PAREN ")"
TOKEN_SEMICOLON ";"

Q: What is a parser?

A: A parser turns the tokens into an abstract syntax tree.

Q: What is an abstract syntax tree (AST)?

A: A hierarchical representation of the source code as a tree of nodes. For example, the function call

trace(i + 1);

is represented by the AST

FunctionCallNode "trace"
  BinaryOperatorPlusNode
    IdentifierNode "i"
    LiteralNode 1

where indentation indicates a parent-child relationship.

Q: What is a definition?

A: A definition is something that can be referred to by name. This includes packages, namespaces, classs, interfaces, functions (including getters and setters), parameters, variables, constants, events, styles, and effects.

Q: What is a scope?

A: A scope corresponding to a curly-brace block in a file (or an entire file) and contains the definitions that occur there. Each definition lives in a particular scope, and scopes can live inside other scopes. In addition, each project has a scope which contains definitions which are visible outside the file in which they occur.

Q: What is name resolution?

A: Name resolution determines which definition an identifier refers to (or "resolves to", or "binds to"). It's what the F2 key does in Flash Builder. For example, in the code

private function f(s:String):void
{
    var i:int = 1;
    g(s, i);
}

the identifier s that is the first argument of the g call resolves to the definition of the parameter named s while the identifier i that is the second argument of the call resolves to the definition of the local variable named i. By resolving an identifier to a definition, the compiler can determine, among other things, the type of the identifier: for example, s is a String, and it can then check whether the definition of the g function accepts a String as its first argument.

Name resolution typically involves searching up the scope chain for a definition with the some name.

Q: What is a code generator?

A: A code generator turns an abstract syntax tree into bytecode.

Q: Where is the specification for ActionScript ByteCode (ABC)?

A: http://www.adobe.com/content/dam/Adobe/en/devnet/actionscript/articles/avm2overview.pdf

Q: What is a semantic checker?

A: A semantic checker finds problems in the source code.

Q: What is an example of how Falcon generates better bytecode than the legacy compiler?

A: Falcon does constant propagation and inlining.

Aditional Information about Falcon and FalconJS

Open Discussion about Falcon and FalconJS
Adobe announces important changes to Flex SDK