Zen and the Art of the Tooling API

Over the Christmas break, I wrote Zen Compiler. It was one part pet project, one part a desire to have Visualforce and Apex being compiled on save while I was tinkering with different editors, and also partly a reason to seriously dive into the Tooling API.

Saleforce introduced the Tooling API as a means to give the community some control over building on the platform without overly relying on a single IDE like Eclipse. Since its release, Eclipse and Sublime+MavensMate have become the general mainstays of developers. Personally I started to gravitate back towards the Developer Console and occasionally using tools like Force CLI when I needed to pull down files or move projects to orgs.

The very idea of developing in multiple languages (Apex, Visualforce, Lightning) without any client side compilation is rather unique to development. When I work in Node, I am mostly doing all of my efforts on localhost. Even that effort doesn't present all errors until run-time, but Editing-Restarting-Debugging-Repeat is significantly faster when you don't have to wait for code changes to be approved by an API.

So Zen had three basic design goals:

  1. Focus solely on code development. Unit testing, project deployment, packaging, logging and other related tasks would be covered by existing tools as needed.

  2. Lightweight. What got me rolling on Zen was realizing that Node's file system could make it possible for an editor agnostic system to just watch for changes and react. For the record, I'm not the first to look at the file system as means to interact with the Tooling API directly, Jason Bury presented a similar idea at Dreamforce and I had the pleasure of being his session manager.

  3. Be fast, flexible and (later while coding it) funny. If you are going to have to wait for your code to compile you might as well get some moderately inspirational phrases along the way.

So how does it work, and possibly more pertinently, how does it work with the Tooling API?

Zen flows like this:

  1. Setup: If you are working with Apex and Visualforce with the Tooling API, the first thing you need is a MetadataContainer. Think of a container like a bucket that you are going to hold your code within. The specific classes, triggers and pages are members (a la an ApexMember) which go into the container. So first (after logging in), Zen either finds or creates a container and any associated member. Typically there won't be an existing member, but since you can only have one member per resource (class, page or trigger) for one container, it is good to check. You can't update your Apex class if you try to create a member instead of updating it.

  2. Wait for file changes and update or create members: Now that we have a container to work with, Zen waits for changes to the directory (and sub directories) it started in. When changes occurs, it assumes the resource type based on the file extension. If it is not a new file, it will get the ID of the resource based on the file name and use that to either update or create a member for it. There is a delay before the next step in case several files are saved at once or nearly at the same time. If that occurs, it will keep updating/adding members to the container until the delay is satisfied. This allows us to bundle as many resources into a compile request at the same time as possible.

  3. Create a Container Request: A ContainerRequest tells the Tooling API that you are done adding or updating members to the container and you are ready to compile. Once that request is sent, every member in the container is locked until the response is returned. However, remember that this is an asynchronous request ... and therefore we need to ...

  4. Poll for the Request: When you create the request you'll get an ID, and you can continue to query the API to see the status of that request. The status will be:

    • Queued—the job is in the queue.
    • Invalidated—Salesforce cancelled the job because the results might not be valid.
    • Completed—the compilation or deployment finished. The SymbolTable fields for the specified object(s) were successfully updated.
    • Failed—the compilation or deployment failed for the reasons stated in the CompilerError field. Error—an unexpected error occurred.
    • Aborted—use this value to delete a queued deployment.
  5. Respond to the Request Status: If the status is queued, give a moderately inspirational phrase to hold the developer over while waiting. If completed, let them now. If there is an error, report the error.

And that is nearly it except for two notable outliers.

The first big outlier is what happens when the file system changes while a Container Request is in the queue. This means the developer kept changing the file after the request was sent - which is actually crazy easy to do when dealing with a text editor like Atom or Sublime. I hit save all the time just to make sure my changes are kept as opposed to compiling the entire class.

In this instance, I had two choices:

  1. Spin up a new container and send that alongside the current container being processed.

  2. Create a backlog of changes and send those once the current request got out of queue.

I started with #1 and ended with #2. This decision was mostly based on the fact that it was easier and more efficient to throw out the delta of changes while the request was still waiting and simply send the latest version of the resources when the queue was free again.

The second outlier: Lightning Components. I am going to do a follow up post to this, but the short version is that compiling Lightning/Aura is not really that difficult. It's just very different than the old system of using the MetaData API and slightly different from what we see above. Basically you have to think in terms of a directory, not as a file.

Anyway, I hope people enjoy Zen Compiler. I'm likely to use it as a touchstone for other utilities to come in the future.

comments powered by Disqus