Logo
loophole letters

Bending JavaScript with shift-ast

code

March 21, 2022

Let's talk more about abstract syntax trees (ASTs). This time, about those representing JavaScript syntax.

Table of Contents

shift-ast

One of the more recent AST formats for JS is shift-ast. It allows parsing, manipulating and generating JavaScript code.

Parse

We can parse a string into an AST using shift-parser:

import { parseScriptWithLocation } from 'shift-parser';

let { tree } = parseScriptWithLocation(`const x = 'a';`);

which yields:

1
1
{
2
2
  "type": "Script",
3
3
  "directives": [],
4
4
  "statements": [
5
5
    {
6
6
      "type": "VariableDeclarationStatement",
7
7
      "declaration": {
8
8
        "type": "VariableDeclaration",
9
9
        "kind": "const",
10
10
        "declarators": [
11
11
          {
12
12
            "type": "VariableDeclarator",
13
13
            "binding": {
14
14
              "type": "BindingIdentifier",
15
15
              "name": "x"
16
16
            },
17
17
            "init": {
18
18
              "type": "LiteralStringExpression",
19
19
              "value": "a"
20
20
            }
21
21
          }
22
22
        ]
23
23
      }
24
24
    }
25
25
  ]
26
26
}


This is how the AST looks visualized:




Manipulate


To transform code, we can walk over nodes with shift-traverser and construct new nodes using shift-ast:


import { replace } from 'shift-traverser';
import { BindingIdentifier } from 'shift-ast';

let bindingIdentifiers = 0;
tree = replace(tree, {
  enter(node, parent) {
    if (node.type === 'BindingIdentifier') {
      return new BindingIdentifier({
        name: `v${bindingIdentifiers++}`,
      });
    }
    return node;
  },
});



This will rename all variables in the AST to v0, v1, etc, creating the following diff:



Expand 11 lines ...
12
12
            "type": "VariableDeclarator",
13
13
            "binding": {
14
14
              "type": "BindingIdentifier",
15
-
              "name": "x"
15
+
              "name": "v0"
16
16
            },
17
17
            "init": {
18
18
              "type": "LiteralStringExpression",
Expand 8 lines ...


Generate


To go full circle, we can take the manipulated AST and generate code from it:


import generate from 'shift-codegen';

generate(tree);



which creates the following diff:


1
-
const x = 'a';
1
+
const v0="a"


Source Locations


Together with the ast, we will also get source location information for each node:


import { parseScriptWithLocation } from 'shift-parser';

let { tree, locations } = parseScriptWithLocation(`const x = 'a';`);

locations.get(tree.statements.declaration.declarators.binding);



With that, we get the exact location of the binding x in our input code:


{
  "start": {
    "line": 1,
    "column": 6,
    "offset": 6
  },
  "end": {
    "line": 1,
    "column": 7,
    "offset": 7
  }
}



What now?


With this tool, sky is the limit, as we can basically take any valid JavaScript code and transform it to suit our needs.


Some things that we can do, which can't be done with regular javascript:


    

  • Rename variables
    • 

  • Operator overloading
    • 

  • Simplification of user code
    • 

  • ... you name it
    • 



Workflow


A good workflow to find out how you want to manipulate your code is


    

  1. go to astexplorer.net
    2. 

  2. entering your input code and copy AST JSON to file input.json
    3. 

  3. entering your target output code and copy AST JSON to file output.json
    4. 

  4. compare both files with the diff viewer of choice, e.g. compare active file with... in VSCode
    5. 

  5. write the code that creates that diff!
    6. 



Outlook


In a future post, I want to show how I used shift-ast to create some useful transforms for my new pet project strudel.


Until then, cheers 🍷
Felix Roos 2023