Posts in category NetBeans

Progress of Migrating AIOTrade to Scala

Well, I've done most parts of migrating AIOTrade to Scala, not all features return yet. I gain lots of experiences of inter-op between Scala and Java, since AIOTrade has to be integrated into an existed Java framework  NetBeans Platform. And also, whole project is now managed by Maven instead of Ant, which reduces lots of pain of dependencies upon crossing sub-projects.

This project is now hosted on  kenai.com  http://sf.net/projects/humaitrader, you can check out the code to get an overview of how to integrated Maven + Scala + NetBeans Modules. Of course, all were done with NetBeans Scala plugin.

LOC of this project so far:

$ ./cloc.pl --read-lang-def=lang_defs.txt ~/myprjs/aiotrade.kn/opensource/
     677 text files.
     617 unique files.                                          
     154 files ignored.

http://cloc.sourceforge.net v 1.08  T=3.0 s (167.7 files/s, 21373.7 lines/s)
-------------------------------------------------------------------------------
Language          files     blank   comment      code    scale   3rd gen. equiv
-------------------------------------------------------------------------------
Scala               353      7981     16301     27180 x   1.36 =       36964.80
Java                 43      1148       833      6946 x   1.36 =        9446.56
XML                 104       231       389      2414 x   1.90 =        4586.60
Bourne Shell          2        81        81       488 x   3.81 =        1859.28
HTML                  1         7        15        26 x   1.90 =          49.40
-------------------------------------------------------------------------------
SUM:                503      9448     17619     37054 x   1.43 =       52906.64
-------------------------------------------------------------------------------

A screen snapshot:

AIOTrade-100117.png

How to Setup Dependencies Aware Maven Project for Scala

I have to say, maintain a couple of dependent projects via Ant is a headache, just like me and others may have encountered when use NetBeans' Scala plugin to create and maintain plain Scala projects, these projects, are all Ant based. It's difficult to write a generic Ant template to compute the dependencies graph and then choose the best building path. The building process of cross dependent projects becomes very slow because the redundant enter/exit dependent projects building task.

NetBeans has also the best Maven plugin integrated, I decided to give Maven a trying. This was actually my first attempt to create Maven based project(s). I explored the mini way toward a Scala Maven project, and patched Scalac a bit again to get the dependencies aware compiling working.

Now here's an example I'd like to share with you.

Assume we have n sub-projects, with cross dependencies, for instance, 'lib.util' and 'lib.math', 'lib.indicators' etc. The simplest Maven way is to keep a parent project (I call it 'modules' here) which holds all common settings and module(sub-project) names(paths). The directory structure could be like:

modules
|-- pom.xml
|-- lib.util
|   |-- pom.xml
|   `-- src
|       |-- main
|       |   `-- scala
|       |       `-- lib
|       |           `-- util
|       |               `-- App.scala
|       `-- test
|           `-- scala
|               `-- lib
|                   `-- util
|                       `-- AppTest.scala
|-- lib.math
    |-- pom.xml
    `-- src
        |-- main
        |   `-- scala
        |       `-- lib
        |           `-- math
        |               `-- App.scala
        `-- test
            `-- scala
                `-- lib
                    `-- math
                        `-- AppTest.scala

What I like Maven here is that the parent project 'modules' only maintains the common settings and module paths, all dependencies between sub-projects are then set in sub-project's self pom.xml file. This is a good decoupled strategy in my eyes, you can freely combine these sub-projects at any time without change too much, and each sub-project doesn't need to care about the dependencies of other projects. The parent project is only a centre place for sharing common setting and a centre place to list all available projects and their directories, it's just like a Directory Service.

Now, here is the pom.xml of parent project looks like (all common settings include compiler, repository etc are kept in it):

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>org.aiotrade</groupId>
    <artifactId>modules</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <name>AIOTrade Modules</name>
    <description>Basic Modules for AIOTrade.</description>

    <properties>
        <scala.version>2.8.0-SNAPSHOT</scala.version>
    </properties>

    <repositories>
        <repository>
            <id>scala-tools.release</id>
            <name>Scala-Tools Release Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
        <repository>
            <id>scala-tools.snapshot</id>
            <name>Scala-Tools Snapshot Repository</name>
            <url>http://scala-tools.org/repo-snapshots</url>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.4</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.specs</groupId>
            <artifactId>specs</artifactId>
            <version>1.2.5</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>process-resources</phase> <!-- to support mix java/scala only -->
                        <goals>
                            <goal>add-source</goal>      <!-- to support mix java/scala only -->
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                    <args>
                        <arg>-target:jvm-1.5</arg>
                        <arg>-make:transitivenocp</arg>
                        <arg>-dependencyfile</arg>
                        <arg>${project.build.directory}/.scala_dependencies</arg>
                    </args>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.5</source>
                    <target>1.5</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <reporting>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
        </plugins>
    </reporting>
    
    <modules>
        <module>lib.math</module>
        <module>lib.util</module>
        <module>lib.indicator</module>
    </modules>
</project>

And the pom.xml of sub-project 'lib.util' looks like:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.aiotrade</groupId>
        <artifactId>modules</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <artifactId>lib-util</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>lib-util</name>

</project>

The 'lib.math' one:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.aiotrade</groupId>
        <artifactId>modules</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <artifactId>lib-math</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>lib-math</name>
    
    <dependencies>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>lib.util</artifactId>
            <version>${project.version}</version>
            <type>jar</type>
        </dependency>
    </dependencies>

</project>

That's all the settings, clear and clean.

Now, let's back to another level of dependencies: dependencies between Scala source files. How does this setting get scalac aware dependencies of changed sources?

Again, just like the scalac setting in an Ant based project, where at 'configuration' part of build/plugins/plugin/@maven-scala-plugin, I add "-make:transitivenocp -dependencyfile ${project.build.directory}/.scala_dependencies" to 'args' (should be put in separate 'arg' tag as following)

                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                    <args>
                        <arg>-target:jvm-1.5</arg>
                        <arg>-make:transitivenocp</arg>
                        <arg>-dependencyfile</arg>
                        <arg>${project.build.directory}/.scala_dependencies</arg>
                    </args>
                </configuration>

Now the last question, how to use NetBeans to create above setting?

Open "File" | "New Project..." | "Maven" | "Maven Project", click "Next", choose "Maven Quickstart Archetype" to create each empty project, then copy/paste above content to corresponding pom.xml, change project groupId, artifactId and version properties. For existed project, copy your source code to "main/scala". You can also mixed Java code, put then at "main/java", NetBeans Scala plugin supports mixed Scala/Java project well.

I'll put the pre-defined archetype on net later, so no need to copy/paste, or, you can create an empty parent/sub-project, commit to your version control system. BTW, I find manually modify pom.xml is a pleasure work.

  • Note No. 1: You have to use/update-to Scala-2.8.0-SNAPHOST with at least scala-compiler-2.8.0-20091128-xxx.jar
  • Note No. 2: If your project is simple, just use NetBeans created Ant based project. If you separate it to a couple of dependent projects, I strongly suggest you to Maven.
  • Note No. 3: A new Scala plugin for NetBeans 6.8RC1 will be available before or at Monday, it's also a must upgrade especially for Windows users.

Scala Plugin for NetBeans - Available for NetBeans 6.8beta and Scala 2.8.0 Snapshot

I packed another release of Scala plugin for NetBeans 6.8 beta, for more information, please see:

 http://wiki.netbeans.org/Scala68v1

This version allows adding "deprecation", "unchecked" parameters for scalac, and fixed unicode id issue. It works on Scala-2.8.0 snapshot only.

Note:

All projects created via "File" -> "New Project" before this version should add or change following lines in "nbproject/project.properties":

scalac.compilerargs=
scalac.deprecation=no
scalac.unchecked=no

Scala Plugin for NetBeans - Available for NetBeans 6.8m2 and Scala 2.8.0 Snapshot

As NetBeans 6.8 m2 released, I packed a downloadable binary which works with Scala 2.8.0.r18993 or above.

Please do not install Scala plugin via Update Center for NetBeans 6.8 m2, it's not compatible. Use the link below to download and install

New & Noteworthy

  • Much more responsive when typing and for code-completion
  • More refactoring: find usages, rename across opened projects
  • Breakpoint works everywhere (almost)
  • Better supporting for mixed Java/Scala project in both direction (Java is visible in Scala and vice versa)
  • Better integration with NetBeans maven plugin
  • Better code completion even for implicit methods (in most cases)
  • Better indentation and formatter for unfinished line after 'if', 'else', '=', 'for', 'while', 'do' etc
  • Better syntax highlighting for val/var, lazy val, implicit call, byname param, abstract method etc
  • Mark of implemented/overridden methods, click on mark will jump to super definition
  • Go to type ("Ctrl+O")
  • Select parts of code, press '{', '[', '(', '"', '`' will add/replace surrounding pair, press '~' will remove them. Specially, press '/' will block-comment it
  • Reset Scala interactive parser when necessary, for instance: dependent libs changed (Right click on source, choose "Reset Scala Parser" in pop-up menu)
  • Output highlighted code to html ([File] -> [Print to HTML...])
  • Some basic hints, for instance: fixing import, unused imports
  • Code template now works, check or add your own via [Options/Preferences] -> [Editor] -> [Code Templates] -> [Scala]

Install with NetBeans 6.8 M2

  1. Download and install the latest Scala-2.8.0 snapshot runtime via Scala's home
  2. Set $SCALA_HOME environment variable to point to the installed Scala runtime path. Add $SCALA_HOME/bin to PATH environment variable. Note for Mac OS user, $SCALA_HOME environment variable may not be visible for Applications/NetBeans, see  http://wiki.netbeans.org/MacOSXEnvForApp
  3. Get the NetBeans 6.8 M2 from:  http://bits.netbeans.org/netbeans/6.8/m2/
  4. Get the Scala plugins binary from:  https://sourceforge.net/projects/erlybird/files/nb-scala/6.8v1.1.0m2/nb-scala-6.8v1.1.0m2.zip/download
  5. Unzip Scala plugin binary to somewhere
  6. Open NetBeans, go to "Tools" -> "Plugins", click on "Downloaded" tab title, click on "Add Plugins..." button, choose the directory where the Scala plugins are unzipped, select all listed *.nbm files, following the instructions.

Tips

  • If you encounter "... Could not connect to compilation daemon.", try to run "fsc -reset" under a command/terminal window.
  • Editor $NetBeansInstallationPath?/etc/netbeans.conf, remove "-ea" to avoid AssertionError? popped up

 http://wiki.netbeans.org/Scala68v1 for more information

Scala Plugin for NetBeans - Rewrite in Scala #8: Partly Visible to Java and Go to Type

>>> Updated on Sep 8:

Now in most cases, Scala is visible to Java, including generic type. Mixed Java/Scala project should be well supported.

======

I need "Go To Type" working, and Scala's symbols should be visible to Java sources. For NetBeans, that means I should implement a Scala to Java VirtualSourceProvider?, which will translate Scala source to Java stub.

I tried it before, but not so successful, so I disabled it. Finally, I got how it works today, and committed work in part. That is, now, in Scala/Java mixed project, not only Java is visible to Scala source, and also, partly, Scala is visible to Java too.

Another benefit is, when you press "Ctrl + O" or "Command + O", a navigator window will bring you to Type's source file that you are searching.

I'll go on to get Scala -> Java mapping fully works soon.

>>> Updated on Sep 8:

The following issue was fixed in trunk.

======

Warning: If you have not got a NetBeans nightly built before, do not try it until a recent (after Sep 3rd) serious fault fixed.

ScalaEditor-090905.png

Scala Plugin for NetBeans - Rewrite in Scala #7: Mark Override Method and Go to Super def

The new progress is a little mark at the left side bar of editor, showing if a method is overriding a method of base class/trait. Move cursor on this mark will show a tooltip, and, click on this mark will jump to source of super definition.

If a mark of (D) shown and you forget to add "override", you will wait until a whole build process to tell you lacking of "override", so pay attention to it.

Of course, if it's an (I) mark, you do not need to add "override".

ScalaEditor-090904.png

Scala Plugin for NetBeans - Rewrite in Scala #5: Refactoring Step One - Finding usages

I'm working on Refactoring features for NetBeans' Scala plugin, the first step is finding usages of class/method/val etc across project's source files, it works now. The next step will be renaming them.

Following snapshot shows class "ScalaDfn?" are used in 19 places, across 6 Scala sources

ScalaEditor-090902.png

Scala Plugin for NetBeans - Rewrite in Scala #6: Refactoring Step Two - Rename

As I've done most code of Refactoring skeleton, it did not cost me too much time to get renaming feature working (2 hours about). Now, NetBeans' Scala plugin can rename class/method/val etc across project's source files.

Following snapshot shows class "Dog" is going to be renamed to "BigDog?". After the preview, press "Do Refactoring", all things done.

ScalaEditor-090903.png

Scala Plugin for NetBeans - Rewrite in Scala #4: How to Use It to Develop Scala Itself

I begin to patch Scala's new designed compiler interface for IDE to get it work better (with NetBeans, and, other IDEs). I know lamp term may use Eclipse as their IDE to develop Scala itself, but I'm using new written NetBeans plugin. Here's a short tutorial of how to use NetBeans Scala plugin to work with Scala itself

Build Scala-2.8.0 snapshot

First and least, svn check out Scala's trunk source code:

cd ~/myprjs/scala/scala
svn co http://lampsvn.epfl.ch/svn-repos/scala/scala/trunk

Then, make sure your $JAVA_HOME is pointed to a JDK 1.5 instead of 1.6, and <b>clear $SCALA_HOME</b>. "ant dist" to build a fresh Scala distribution, which is located at ~myprjs/scala/scala/dist/latest

# As for Mac OS X
JAVA_HOME_BAK=$JAVA_HOME
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home
export SCALA_HOME=
ant dist
export JAVA_HOME=$JAVA_HOME_BAK
export SCALA_HOME=~myprjs/scala/scala/dist/latest

Install latest NetBeans and Scala plugin

Download latest NetBeans nightly built from  http://bits.netbeans.org/download/trunk/nightly/latest, the minimal pack "Java SE" is enough.

Run NetBeans, get latest Scala plugin via: [Preference/Option] -> [Plugins] -> [Available Plugins], find "Scala Kit" in the list and choose it,following the instructions to get it installed, restart NetBeans.

Create NetBeans project by importing existing sources

Create project for Scala's trunk sources. <b>Note: each folder under "src" should be considered as standalone folder, for example, "compiler"</b>, it's better to create standalone project for each these folder, I created two, one for "compiler",another for "library".

[File] -> [New Project] -> [Scala] -> [Scala Project with Existing Sources]

ScalaTrunkProject1.png

Click [Next], input project name that you like Note: you can input an existed ant build script file name in "Build Script Name". Or, modify the auto-created one later, it's a plain ant build xml file.

ScalaTrunkProject2.png

Click [Next], add source folder:

ScalaTrunkProject2.png

Click [Next] again, you can see the sources that will be included in this project, then click [Finish]

ScalaTrunkProject4.png

Now, Scala's compiler has been imported as a NetBeans freeform project, it is an old plain ant project. You can create another one for Scala's library.

ScalaTrunkProject5.png

How to debug into Scala's compiler, library source?

If you have a project that want to debug into Scala's compiler, library sources, you could do when in debugging:

In debugging, open debugging source windows via: [Windows] -> [Debugging] -> [Sources]. Go through the listed source jars, find ".../scala-compiler.jar" and ".../scala-library.jar", check them.

Scala Plugin for NetBeans - Rewrite in Scala #3: Ready for Beta Test

I struggled with  new redesigned IDE / compiler interface (interative.Global) during this weekend, and finally, got it attached to NetBeans Scala plugin, which runs as a background daemon thread, thus the plugin is more responsive when coding (the first benefit). It was a hard work though, I had to modify some of the original code to get whole things stable and responsive (balance), it paid me 2 sleepless nights.

So here's what's new on current nightly built:

  • Tested and work with Scala 2.8.0.r18542 snapshot
  • Better supporting for mixed Java/Scala project
  • Better indentation and formatter for unfinished line after 'if', 'else', '=', 'for', 'while', 'do' etc
  • Better code completion even for implicit methods (some times)
  • Implicit method call will be highlighted by underline
  • Select parts of code, press '{', '[', '(', '"', '`' etc will add/replace surrounding pair, press '~' will remove them. Specially, press '/' will block comment it
  • Some basic hints, for example, fix import (from Milos' work)
  • Code template now works, check or add your own via [Options/Preferences] -> [Editor] -> [Code Templates] -> [Scala] (from Milos' work)

Note: wait until Monday night for NetBeans' hudson builds those whole new nbms.

I think this plugin is qualifiable for beta using/testing now. If you are interested in testing/reporting bugs, you can get the latest NetBeans nightly built, and got the new Scala plugin from Update Center, download a Scala-2.8.0 snapshot, live your $SCALA_HOME to this Scala 2.8.0 snapshot, change $PATH to including $SCALA_HOME/bin.

By testing/using Scala 2.8.0 right now, we can also push it improved more quickly.

BTW, I'm using this plugin on whole Scala' trunk source and of course, this plugin.

Scala Plugin for NetBeans - Rewrite in Scala #2: Supports Java/Scala Mixed Project

Java/Scala mixed project was supported partly once before, but I cannot remember when this feature got lost. Anyway, as one progressing of the rewritten NetBeans Scala plugin, I just got this feature working properly halfway: now Java source is visible to Scala one, but not vice versa. To got Scala source also visible for Java source, would pay for efficiency, since the compiler may need to be pushed to a deep phase, and mapped to Java classes/methods, I need more time to think about that.

Below is a screenshot that a JavaDog? is called from a Scala source file:

ScalaEditor-090810.png

Scala Plugin for NetBeans - Rewrite in Scala #1: Almost Done

The previous Scala plugin for NetBeans was a rush work which was written in Java, toward a useful tool to support writing Scala code. I called it chicken, which was then can be used to produce new Scala plugin written in Scala, that was, the egg.

The egg has grown to a chicken now. As for last night, I switched whole scala modules to depend on the new written scala.editor module (in Scala), and use this new chicken to improve Scala plugin from now on.

The new scala plugin will be carefully rewritten, with a lot of APIs supporting for language editor in mind, and try to support better and more features toward a whole featured, stable Scala IDE.

Another good news is, Milos Kleint, the contributor to NetBeans Maven plugin, has also put hands on this project, he's working on better Scala support for maven project, and new templates/hinds features for Scala editor modules.

The new plugin is based on Scala 2.8 snapshot, if want to get it, you should use the nightly built NetBeans from netbeans.org, and get the new plugins via updated center.

Note: you have to install the latest Scala 2.8 snapshot too, and make sure $SCALA_HOME, $PATH pointed to it. For maven project, you should also update your pom.xml to depend on Scala -2.8-snapshot.

Below is a working snapshot that I was using new Scala plugin to write Scala plugin:

ScalaEditor-090808.png

Scala Plugin for NetBeans: Scala 2.8 Snapshot

>>> Updated

Don't forget to download a fresh Scala 2.8.0-snapshot from scala-lang.org (you may want to track the progress of 2.8.0 once a week etc), and set SCALA_HOME, PATH environment to point to the installation of it.

======

During the weekend, I built a fresh Scala 2.8.0 snapshot. With  ticket #2163 fixed in 18 hours (great job, Scala team), I got a Scala plugin for NetBeans that works under Scala 2.8.0. I then re-built whole Erlang plugin (which is written in Scala), by fixing some minor incompatible issues, it works too.

So, at least, Scala 2.8 seems stable enough for writing a NetBeans plugin now.

Here's a snapshot showing some new features of 2.8.0

ScalaEditor-090720.png

If you are eager to have some taste of Scala 2.8.0 with NetBeans plugin, and is patient to keep another NetBeans nightly version on your machine, then, wait for one more day from now for NetBeans hudson to build whole things, download the lasted NetBeans nightly version, and install "Scala Kit" module via "Update Center".

Don't forget to download a fresh Scala 2.8.0-snapshot (you may want to track the progress of 2.8.0) from scala-lang.org, and maintain your SCALA_HOME, and PATH environment for different Scala versions

BTW, if I have time, I'll begin to rewrite this plugin in Scala, on 2.8.0

Rats! Plugin for NetBeans#1: Syntax Highlighting

I've used  Rats! parser generator heavily on Scala/Erlang plugin for NetBeans, but not write a plugin for Rats! itself.

So I spent my weekend days on a simple Rats! editor module, which implemented syntax highlighting. It's built on Scala.

Here's the snapshot:

ratsEditor-090630.png

It will be available in couple of days.

Erlang Plugin Version 1 for NetBeans 6.7 Released

I'm pleased to announce Erlang plugin (ErlyBird) version 1 for NetBeans 6.7 is released.

NetBeans 6.7 RC3 or above is a requirement.

What's new:

  • It's rewritten in Scala instead of Java
  • More reliable instant rename
  • Display extracted document information from source comment when doing auto-completion

To download, please go to:  https://sourceforge.net/project/showfiles.php?group_id=192439&package_id=226387

To install:

  • Open NetBeans, go to "Tools" -> "Plugins", click on "Downloaded" tab title, click on "Add Plugins..." button, choose the directory where the Erlang plugin are unzipped, select all listed *.nbm files, following the instructions.
  • Make sure your Erlang bin path is under OS environment PATH, you can also check/set your OTP path: From [Tools]->[Erlang Platform], fill in the full path of your 'erl.exe' or 'erl' file in "Interpreter", for instance: "C:/erl/bin/erl.exe". Or open the "Brows" dialog to locate the erlang installation.

<li>When you open/create an Erlang project first time, the OTP libs will be indexed. Take a coffee and wait, the indexing time varies from 10 to 30 minutes depending on your computer.

Feedback and bug reports are welcome.

Scala Plugin Version 1 for NetBeans 6.7 Released

>>> Updated on July 1, 2009

There are couple of reports on "Could not connect to compilation daemon.", mostly under Windows OS. It's a known issue, to resolve it, please check the following:

  • SCALA_HOME is set to a fresh installed Scala runtime
  • PATH includes $SCALA_HOME/bin
  • If the build task still complain, try to run "fsc" or "scala" in a command window first

======

I'm pleased to announce the availability of Scala plugin version 1 for NetBeans 6.7

What's new:

  • Use fsc instead of scalac as the project building compiler (If you've set SCALA_HOME, make sure $SCALA_HOME/bin is included in your PATH environment).
  • Fixed setting breakpoint in closure statement.
  • A basic import-fixer (Right click on source, then choose "Fix Imports").
  • Code assistant for local vars and functions.
  • Run/Debug single file.

To download, please go to:  https://sourceforge.net/project/showfiles.php?group_id=192439&package_id=256544&release_id=686747

For more information, please see  http://wiki.netbeans.org/Scala

Bug reports are welcome.

It works on NetBeans 6.7 RC1 or above.

Erlang Plugin for NetBeans in Scala#11: Indexer

For IDE support, an indexer is very important for declaration-find, code-completion, re-factory etc. Indexer will monitor the standard lib and your project, analyzing the source/binary files, gather meta-information of classes, functions, global vars, usages, and store them in a search engine (It's a Lucene back-end in NetBeans). That means, you need to support project management and platform management first, so you know where's the class/source path of platform and your current opened projects.

ErlyBird implemented Erlang platform and project management in modules: erlang.platform and erlang.project, these code are derived from Tor's Ruby module, I've migrated them to CSL, but did not rewrote them in Scala. It's OK for integrating these Java written modules with Scala written erlang.editor module.

First, you should identify the class path type, you can register them in ErlangLanguage.scala as:

    /** @see org.netbeans.modules.erlang.platform.ErlangPlatformClassPathProvider and ModuleInstall */
    override
    def getLibraryPathIds = Collections.singleton(BOOT)

    override
    def getSourcePathIds = Collections.singleton(SOURCE)

object ErlangLanguage {
    val BOOT    = "erlang/classpath/boot"
    val COMPILE = "erlang/classpath/compile"
    val EXECUTE = "erlang/classpath/execute"
    val SOURCE  = "erlang/classpath/source"
}

Where, BOOT, thus getLibraryPathIds will point to Erlang OTP libs' path later in:
org.netbeans.modules.erlang.platform.ErlangPlatformClassPathProvider

Now, the indexer itself, ErlangIndex.scala which extends CSL's EmbeddingIndexer and implemented:

override protected def index(indexable:Indexable, parserResult:Result, context:Context) :Unit

with a factory:

class Factory extends EmbeddingIndexerFactory

Register it in ErlangLanguage.scala as:

    override
    def getIndexerFactory = new ErlangIndexer.Factory

Now, tell the CSL to monitor and index them, for Erlang OTP libs, this is done in ModuleInstall.java:

    @Override
    public void restored() {
        GlobalPathRegistry.getDefault().register(ErlangPlatformClassPathProvider.BOOT, new ClassPath[] { ErlangPlatformClassPathProvider.getBootClassPath() });
    }
    
    @Override
    public void uninstalled() {
        GlobalPathRegistry.getDefault().unregister(ErlangPlatformClassPathProvider.BOOT, new ClassPath[] { ErlangPlatformClassPathProvider.getBootClassPath() });
    }

All classpaths registered in GlobalPathRegistry will be monitored and indexed automatically. As "ModuleInstall.java" is also registered in module erlang.platform's manifest.mf as OpenIDE-Module-Install class, it will be automatically loaded and invoked when this module is installed/activated in NetBeans:

Manifest-Version: 1.0
OpenIDE-Module: org.netbeans.modules.erlang.platform
OpenIDE-Module-Layer: org/netbeans/modules/erlang/platform/resources/layer.xml
OpenIDE-Module-Localizing-Bundle: org/netbeans/modules/erlang/platform/Bundle.properties
OpenIDE-Module-Install: org/netbeans/modules/erlang/platform/ModuleInstall.class
OpenIDE-Module-Specification-Version: 0.18.0

For project's classpath, the "GlobalPathRegistry registering is at RubyProject.java (don't be confused by the name, it's a derivation from Ruby's code, I did not change it yet), as:

    private final class ProjectOpenedHookImpl extends ProjectOpenedHook {
        
        ProjectOpenedHookImpl() {}
        
        protected void projectOpened() {
            // register project's classpaths to GlobalPathRegistry
            ClassPathProviderImpl cpProvider = lookup.lookup(ClassPathProviderImpl.class);
            GlobalPathRegistry.getDefault().register(RubyProject.BOOT, cpProvider.getProjectClassPaths(RubyProject.BOOT));
            GlobalPathRegistry.getDefault().register(RubyProject.SOURCE, cpProvider.getProjectClassPaths(RubyProject.SOURCE));            
        }
        
        protected void projectClosed() {
            // unregister project's classpaths to GlobalPathRegistry
            ClassPathProviderImpl cpProvider = lookup.lookup(ClassPathProviderImpl.class);
            GlobalPathRegistry.getDefault().unregister(RubyProject.SOURCE, cpProvider.getProjectClassPaths(RubyProject.SOURCE));
        }
        
    }

So the project's source path will be registered when this project is opened, thus, trigger the index engine.

Now, when you first install Erlang plugin, IDE will index whole OTP libs once. And when an Erlang project is opened, the project's source files will be indexed too.

Then, you need to do a reverse job: search index, get the meta-data to composite Erlang symbols and AST items. This is done in ErlangIndex.scala, which has helper methods to search the index fields, and fetch back the stored signature string, extract it to symbol, and invoke the parsing manager to resolve the AST items. With the help of ErlangIndex.scala, you can search across the whole libs, find completion items, find declarations, find documents embedded in source comments, find usages.

As the first use of index feature, I've implemented the code-completion and go-to-declaration for global modules functions:

nn

When user input "lists:", will invoke code-completion feature, since it's a remote function call, code-completion will search the index data, and find it's from OTP's "lists.erl", after parsing this file, we get all exported functions AstDfn, so, there are a lot of meta-information can be used now, I'll implement the documents tooltips later.

This will be the final one of the series of Erlang plug-in in Scala blogs, thanks for reading and feedback. I'll take a break during the weekend. From the next Monday, with the spring is coming, I'll join a new project, which is an ambitious financial information platform, I'll bring Scala, Lift, Erlang to it. BTW, congratulations to Lift 1.0!

Erlang Plugin for NetBeans in Scala#10: Code Completion

Implementing Code-Completion is a bit complex, but you can got it work gradually. At the first step, you can implement Code-Completion for local vars/functions only, then, with the indexed supporting, you can add completion for remote functions.

You should define some kinds of completion proposal, which may show different behaviors when they are popped up and guard you followed steps. For example, a function proposal can auto-fill parameters, on the other side, a keyword proposal just complete itself.

The completion proposal classes are defined in  ErlangComplectionProposal.scala, which implemented CSL's interface CompletionProposal. you may notice that the function proposal is the most complex one, which should handle parameters information.

Then, you should implement CSL's interface CodeCompletionHandler, for Erlang, it's  ErlangCodeCompletion, where, the key method is:

    override
    def complete(context:CodeCompletionContext) :CodeCompletionResult = {
        this.caseSensitive = context.isCaseSensitive
        val pResult = context.getParserResult.asInstanceOf[ErlangParserResult]
        val lexOffset = context.getCaretOffset
        val prefix = context.getPrefix match {
            case null => ""
            case x => x
        }

        val kind = if (context.isPrefixMatch) QuerySupport.Kind.PREFIX else QuerySupport.Kind.EXACT
        val queryType = context.getQueryType

        val doc = LexUtil.document(pResult, true) match {
            case None => return CodeCompletionResult.NONE
            case Some(x) => x.asInstanceOf[BaseDocument]
        }

        val proposals = new ArrayList[CompletionProposal]
        val completionResult = new DefaultCompletionResult(proposals, false)

        // Read-lock due to Token hierarchy use
        doc.readLock
        try {
            val astOffset = LexUtil.astOffset(pResult, lexOffset)
            if (astOffset == -1) {
                return CodeCompletionResult.NONE
            }
            val root = pResult.rootScope match {
                case None => return CodeCompletionResult.NONE
                case Some(x) => x
            }
            val th = LexUtil.tokenHierarchy(pResult).get
            val fileObject = LexUtil.fileObject(pResult).get

            val request = new CompletionRequest
            request.completionResult = completionResult
            request.result = pResult
            request.lexOffset = lexOffset
            request.astOffset = astOffset
            request.index = ErlangIndex.get(pResult)
            request.doc = doc
            request.info = pResult
            request.prefix = prefix
            request.th = th
            request.kind = kind
            request.queryType = queryType
            request.fileObject = fileObject
            request.anchor = lexOffset - prefix.length
            request.root = root
            ErlangCodeCompletion.request = request
            
            val token = LexUtil.token(doc, lexOffset - 1) match {
                case None => return completionResult
                case Some(x) => x
            }

            token.id match {
                case ErlangTokenId.LineComment =>
                    // TODO - Complete symbols in comments?
                    return completionResult
                case ErlangTokenId.StringLiteral =>
                    //completeStrings(proposals, request)
                    return completionResult
                case _ =>
            }
            
            val ts = LexUtil.tokenSequence(th, lexOffset - 1) match {
                case None => return completionResult
                case Some(x) =>
                    x.move(lexOffset - 1)
                    if (!x.moveNext && !x.movePrevious) {
                        return completionResult
                    }
                    x
            }
 
            val closetToken = LexUtil.findPreviousNonWsNonComment(ts)

            if (root != null) {
                val sanitizedRange = pResult.sanitizedRange
                val offset = if (sanitizedRange != OffsetRange.NONE && sanitizedRange.containsInclusive(astOffset)) {
                    sanitizedRange.getStart
                } else astOffset

                val call = Call(null, null, false)
                findCall(root, ts, th, call, 0)
                val prefixBak = request.prefix
                call match {
                    case Call(null, _, _) =>
                    case Call(base, _, false) =>
                        // it's not a call, but may be candicate for module name, try to get modules and go-on
                        completeModules(base, proposals, request)
                    case Call(base, select, true) =>
                        if (select != null) {
                            request.prefix = call.select.text.toString
                        } else {
                            request.prefix = ""
                        }
                        completeModuleFunctions(call.base, proposals, request)
                        // Since is after a ":", we won't added other proposals, just return now whatever
                        return completionResult
                }
                request.prefix = prefixBak
                completeLocals(proposals, request)
            }

            completeKeywords(proposals, request)
        } finally {
            doc.readUnlock
        }

        completionResult
    }

For a Erlang function call, you should check the tokens surrounding the caret to get the call's base name and select first, which is done by a method findCall:

    private def findCall(rootScope:AstRootScope, ts:TokenSequence[TokenId], th:TokenHierarchy[_], call:Call, times:Int) :Unit = {
        assert(rootScope != null)
        val closest = LexUtil.findPreviousNonWsNonComment(ts)
        val idToken = closest.id match {
            case ErlangTokenId.Colon =>
                call.caretAfterColon = true
                // skip RParen if it's the previous
                if (ts.movePrevious) {
                    val prev = LexUtil.findPreviousNonWs(ts)
                    if (prev != null) {
                        prev.id match {
                            case ErlangTokenId.RParen   => LexUtil.skipPair(ts, ErlangTokenId.LParen,   ErlangTokenId.RParen,   true)
                            case ErlangTokenId.RBrace   => LexUtil.skipPair(ts, ErlangTokenId.LBrace,   ErlangTokenId.RBrace,   true)
                            case ErlangTokenId.RBracket => LexUtil.skipPair(ts, ErlangTokenId.LBracket, ErlangTokenId.RBracket, true)
                            case _ =>
                        }
                    }
                }
                LexUtil.findPrevIncluding(ts, LexUtil.CALL_IDs)
            case id if LexUtil.CALL_IDs.contains(id) => closest
            case _ => null
        }

        if (idToken != null) {
            times match {
                case 0 if call.caretAfterColon => call.base = idToken
                case 0 if ts.movePrevious => LexUtil.findPreviousNonWsNonComment(ts) match {
                        case null => call.base = idToken
                        case prev if prev.id == ErlangTokenId.Colon =>
                            call.caretAfterColon = true
                            call.select = idToken
                            findCall(rootScope, ts, th, call, times + 1)
                        case _ => call.base = idToken
                    }
                case _ => call.base = idToken
            }
        }
    }

    case class Call(var base:Token[TokenId], var select:Token[TokenId], var caretAfterColon:Boolean)

To complete a remote function call, you may need to visit outer modules, which needs an indexer, so as the first step, you can just ignore it, go straight to complete local vars/functions, or keywords:

    private def completeLocals(proposals:List[CompletionProposal], request:CompletionRequest) :Unit = {
        val prefix = request.prefix
        val kind = request.kind
        val pResult = request.result

        val root = request.root
        val closestScope = root.closestScope(request.th, request.astOffset) match {
            case None => return
            case Some(x) => x
        }
        val localVars = closestScope.visibleDfns(ElementKind.VARIABLE)
        localVars ++= closestScope.visibleDfns(ElementKind.PARAMETER)
        localVars.filter{v => filterKind(kind, prefix, v.name)}.foreach{v =>
            proposals.add(new PlainProposal(v, request.anchor))
        }

        val localFuns = closestScope.visibleDfns(ElementKind.METHOD)
        localFuns.filter{f => filterKind(kind, prefix, f.name)}.foreach{f =>
            proposals.add(new FunctionProposal(f, request.anchor))
        }
    }

    private def completeKeywords(proposals:List[CompletionProposal], request:CompletionRequest) :Unit = {
        val prefix = request.prefix
        val itr = LexerErlang.ERLANG_KEYWORDS.iterator
        while (itr.hasNext) {
            val keyword = itr.next
            if (startsWith(keyword, prefix)) {
                proposals.add(new KeywordProposal(keyword, null, request.anchor))
            }
        }
    }

There is a function def visibleDfns(kind: ElementKind): ArrayBuffer[AstDfn] in AstRootScope.scala, if you've put definition items properly in scopes, it should handle the visibility automatically.

Now, register it in  ErlangLanguage.scala:

    override
    def getCompletionHandler = new ErlangCodeCompletion

As usual, run it, you got:

ErlangEditor-090227.png

The local functions and vars are proposed plus the keywords. BTW, I've fixed this feature for Scala plug-in.

Better Look&Feel of NetBeans on Mac OS

NetBeans 6.7M2 is going to be public available soon, the new look&feel on Mac OS is very awesome. BTW, -Dapple.awt.graphics.UseQuartz=true is a must option for best font-rendering on my macbook. I added it to netbeans.conf as -J-Dapple.awt.graphics.UseQuartz=true. Here is a snapshot of my current screen when working on Erlang plugin in Scala:

Click on the picture to enlarge it:

NetBeans-macos-090222.png

Erlang Plugin for NetBeans in Scala#9: Instant Rename and Go to Declaration

It seems I'm the only one who is committing to hg.netbeans.org/main/contrib these days. Anyway.

Implementing Instant Rename and Go to Declaration is not difficult with AstDfn/AstRef/AstScope. Here is the code:

And register them in :

    override
    def getInstantRenamer = new ErlangInstantRenamer

    override
    def getDeclarationFinder = new ErlangDeclarationFinder

Only local function definitions can be traveled to at this time. To jump to remote function definitions, I have to implement an indexer first.

Erlang Plugin for NetBeans in Scala#8: Pretty Formatting and Pair Matching

Now let's go on the complex part: Pretty Formatting and Pair Matching. I say they are complex, not because these features are much heavier on language's semantic complex. Implementing these features mostly deals with lexer tokens. But it's a bit brain-dried work to across forward/backward in the token stream to get the pair matching and pretty formatting working as you expected.

Because of the complex, I won't describe the details of how to implement them for Erlang, I just put the links to these source code:

And registered them in ErlangLanguage.scala as:

    override
    def getKeystrokeHandler = new ErlangKeystrokeHandler

    override
    def hasFormatter =  true

    override
    def getFormatter = new ErlangFormatter

With these feature implemented, the new plugin can automatically complete/match braces and pair, indent properly when you hit BREAK, input a "end" etc.

BTW, the navigator window was improved during these days, it can now properly show the arity/args of each functions. It's done by improved AstNodeVisitor.scala and AstDfn.scala

nn

Erlang Plugin for NetBeans in Scala#7: Occurrences Finder

During the AST node visiting (AstNodeVisitor.scala), I've gathered a lot of variable definitions and their references, we can now try to get editor to mark these occurrences. There are preliminary functions in AstScope.scala, such as findOccurrences(AstItem). You can override AstDfn#isReferredBy(AstRef) and AstRef#isOccurence(AstRef) to get accurate reference relations. As the first step, I just simply judge the reference relation by the equation of names.

We'll extends org.netbeans.modules.csl.api.OccurrencesFinder and implement:

run(pResult:ErlangParserResult, event:SchedulerEvent) :Unit

First, we try to get the AstItem which is at the caretPosition by rootScope.findItemAt(th, caretPosition), and verify if it's a valid token.

Then, by calling rootScope.findOccurrences(item), we collect all occurrences of this item, and put a ColoringAttributes.MARK_OCCURRENCES with its OffsetRange.

The code of ErlangOccurrencesFinder.scala:

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 *
 * Contributor(s):
 *
 * Portions Copyrighted 2009 Sun Microsystems, Inc.
 */
package org.netbeans.modules.erlang.editor

import _root_.java.util.{HashMap,List,Map}
import javax.swing.text.Document
import org.netbeans.api.lexer.{Token,TokenId,TokenHierarchy}
import _root_.org.netbeans.modules.csl.api.{ColoringAttributes,OccurrencesFinder,OffsetRange}
import org.netbeans.modules.parsing.spi.Parser
import org.netbeans.modules.parsing.spi.{Scheduler,SchedulerEvent}
import org.netbeans.modules.erlang.editor.ast.{AstDfn,AstItem,AstRef,AstRootScope}
import org.netbeans.modules.erlang.editor.lexer.{ErlangTokenId,LexUtil}
import org.openide.filesystems.FileObject

/**
 *
 * @author Caoyuan Deng
 */
class ErlangOccurrencesFinder extends OccurrencesFinder[ErlangParserResult] {

    protected var cancelled = false
    private var caretPosition = 0
    private var occurrences :Map[OffsetRange, ColoringAttributes] = _
    private var file :FileObject = _

    protected def isCancelled = synchronized {cancelled}

    protected def resume :Unit = synchronized {cancelled = false}

    override
    def getPriority = 0

    override
    def getSchedulerClass = Scheduler.CURSOR_SENSITIVE_TASK_SCHEDULER

    override
    def getOccurrences :Map[OffsetRange, ColoringAttributes] = occurrences

    override
    def cancel :Unit = synchronized {cancelled = true}

    override
    def setCaretPosition(position:Int) :Unit = {this.caretPosition = position}

    def run(pResult:ErlangParserResult, event:SchedulerEvent) :Unit = {
        resume
        if (pResult == null || isCancelled) {
            return
        }

        val currentFile = pResult.getSnapshot.getSource.getFileObject
        if (currentFile != file) {
            // Ensure that we don't reuse results from a different file
            occurrences = null
            file = currentFile
        }

        for (rootScope <- pResult.rootScope;
             th <- LexUtil.tokenHierarchy(pResult);
             doc <- LexUtil.document(pResult, true);
             // * we'll find item by offset of item's idToken, so, use caretPosition directly
             item <- rootScope.findItemAt(th, caretPosition);
             idToken <- item.idToken
        ) {
            var highlights = new HashMap[OffsetRange, ColoringAttributes](100)

            val astOffset = LexUtil.astOffset(pResult, caretPosition)
            if (astOffset == -1) {
                return
            }

            // * When we sanitize the line around the caret, occurrences
            // * highlighting can get really ugly
            val blankRange = pResult.sanitizedRange

            if (blankRange.containsInclusive(astOffset)) {
                return
            }

            // * test if document was just closed?
            LexUtil.document(pResult, true) match {
                case None => return
                case _ =>
            }

            try {
                doc.readLock
                val length = doc.getLength
                val astRange = LexUtil.rangeOfToken(th.asInstanceOf[TokenHierarchy[TokenId]], idToken.asInstanceOf[Token[TokenId]])
                val lexRange = LexUtil.lexerOffsets(pResult, astRange)
                var lexStartPos = lexRange.getStart
                var lexEndPos = lexRange.getEnd

                // If the buffer was just modified where a lot of text was deleted,
                // the parse tree positions could be pointing outside the valid range
                if (lexStartPos > length) {
                    lexStartPos = length
                }
                if (lexEndPos > length) {
                    lexEndPos = length
                }

                LexUtil.token(doc, caretPosition) match {
                    case None => return
                    case token => // valid token, go on
                }
            } finally {
                doc.readUnlock
            }

            val _occurrences = rootScope.findOccurrences(item)
            for (_item <- _occurrences;
                 _idToken <- _item.idToken
            ) {
                highlights.put(LexUtil.rangeOfToken(th.asInstanceOf[TokenHierarchy[TokenId]],
                                                    _idToken.asInstanceOf[Token[TokenId]]),
                               ColoringAttributes.MARK_OCCURRENCES)
            }

            if (isCancelled) {
                return
            }

            if (highlights.size > 0) {
                val translated = new HashMap[OffsetRange, ColoringAttributes](2 * highlights.size)
                val entries = highlights.entrySet.iterator
                while (entries.hasNext) {
                    val entry = entries.next
                    LexUtil.lexerOffsets(pResult, entry.getKey) match {
                        case OffsetRange.NONE =>
                        case range => translated.put(range, entry.getValue)
                    }
                }

                highlights = translated

                this.occurrences = highlights
            } else {
                this.occurrences = null
            }
        }
    }
}

Finally, again, register it in ErlangLanguage.scala

override def hasOccurrencesFinder = true
override def getOccurrencesFinder = new ErlangOccurrencesFinder

And add mark-occurrences effect setting in fontsColors.xml:

<fontcolor name="mark-occurrences" bgColor="ECEBA3"/>

That's all. Run it, you got:

nn

Var LogPid's definition and references were highlighted now.

The correctness of occurrences marking depends on the correctness of scopes of variables when you visit AST node, I did not finish all of such work yet, but, enabling occurrences marking feature is very helpful for further work.

Erlang Plugin for NetBeans in Scala#6: Semantic Analyzer

With more detailed AstNodeVisitor.scala, I got the semantic information of function calls, variable definitions and references etc. It's time to implement CSL's SemanticAnalyzer, which is the entrance of semantic highlighting.

I then encountered a Scala's corner case issue :-(

SemanticAnalyzer.java is a sub-class of ParserResultTask:

package org.netbeans.modules.parsing.spi;
public abstract class ParserResultTask<T extends Parser.Result> extends SchedulerTask {
    public abstract void run (T result, SchedulerEvent event);
    public abstract int getPriority ();
}

ParserResultTask's signature in class file is:

<<TLorg/netbeans/modules/parsing/spi/Parser$Result;>Lorg/netbeans/modules/parsing/spi/SchedulerTask;>

run's method signature:

<(TT;Lorg/netbeans/modules/parsing/spi/SchedulerEvent;)V>

ErlangSemanticAnalyzer extended SemanticAnalyzer, so I should implement:

void run (T result, SchedulerEvent event);

which carries type parameter T from ParserResultTask<T extends Parser.Result>. But unfortunately, SemanticAnalyzer extends ParserResultTask as:

public abstract class SemanticAnalyzer extends ParserResultTask;

That is, SemanticAnalyzer erased type parameter of its parent. It's valid in Java. But in Scala, I can not successfully extend SemanticAnalyzer anymore, since when I tried to implement "void run (T result, SchedulerEvent event)", the scalac always complained that I did not override this method with type T. It seems scalac checks type parameter deeply, and needs sub-class to always explicitly extend parent class with type parameter. Since SemanticAnalyzer has erased type parameter, even I tried to implement "run" as:

override def run[T <: Parser.Result](result:T, event:SchedulerEvent) :Unit 

scalac still complained about it.

I have no idea of how to bypass this issue. I then patched SemanticAnalyzer.java, let it carries same type parameter as its parent:

public abstract class SemanticAnalyzer<T extends Parser.Result> extends ParserResultTask<T> 

Now everything works. My final code is:

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2008 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 *
 * Contributor(s):
 *
 * Portions Copyrighted 2009 Sun Microsystems, Inc.
 */
package org.netbeans.modules.erlang.editor

import _root_.java.util.{HashMap,HashSet,Map,Set}
import javax.swing.text.Document
import org.netbeans.api.lexer.Token
import org.netbeans.api.lexer.TokenHierarchy
import org.netbeans.api.lexer.TokenId
import org.netbeans.modules.csl.api.{ColoringAttributes,ElementKind,OffsetRange,SemanticAnalyzer}
import org.netbeans.modules.parsing.spi.Parser
import org.netbeans.modules.parsing.spi.{Scheduler,SchedulerEvent,ParserResultTask}
import org.netbeans.modules.erlang.editor.ast.{AstDfn,AstItem,AstRef,AstRootScope}
import org.netbeans.modules.erlang.editor.lexer.LexUtil
import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId

/**
 *
 * @author Caoyuan Deng
 */
class ErlangSemanticAnalyzer extends SemanticAnalyzer[ErlangParserResult] {

    private var cancelled = false
    private var semanticHighlights :Map[OffsetRange, Set[ColoringAttributes]] = _

    protected def isCancelled :Boolean = synchronized {cancelled}

    protected def resume :Unit = synchronized {cancelled = false}

    override
    def getHighlights :Map[OffsetRange, Set[ColoringAttributes]] = semanticHighlights

    override
    def getPriority = 0

    override
    def getSchedulerClass = Scheduler.EDITOR_SENSITIVE_TASK_SCHEDULER
    
    override
    def cancel :Unit = {cancelled = true}

    @throws(classOf[Exception])
    override
    def run(pResult:ErlangParserResult, event:SchedulerEvent) :Unit = {
        resume
        semanticHighlights = null

        if (pResult == null || isCancelled) {
            return
        }

        for (rootScope <- pResult.rootScope;
             th <- LexUtil.tokenHierarchy(pResult);
             doc <- LexUtil.document(pResult, true)
        ) {
            var highlights = new HashMap[OffsetRange, Set[ColoringAttributes]](100)
            visitItems(th.asInstanceOf[TokenHierarchy[TokenId]], rootScope, highlights)

            this.semanticHighlights = if (highlights.size > 0) highlights else null
        }
    }

    private def visitItems(th:TokenHierarchy[TokenId], rootScope:AstRootScope, highlights:Map[OffsetRange, Set[ColoringAttributes]]) :Unit = {
        import ElementKind._
        for (item <- rootScope.idTokenToItem(th).values;
             hiToken <- item.idToken
        ) {
            
            val hiRange = LexUtil.rangeOfToken(th, hiToken.asInstanceOf[Token[TokenId]])
            item match {
                case dfn:AstDfn => dfn.getKind match {
                        case MODULE =>
                            highlights.put(hiRange, ColoringAttributes.CLASS_SET)
                        case CLASS =>
                            highlights.put(hiRange, ColoringAttributes.CLASS_SET)
                        case ATTRIBUTE =>
                            highlights.put(hiRange, ColoringAttributes.STATIC_SET)
                        case METHOD =>
                            highlights.put(hiRange, ColoringAttributes.METHOD_SET)
                        case PARAMETER =>
                            highlights.put(hiRange, ColoringAttributes.PARAMETER_SET)
                        case _ =>
                    }
                case ref:AstRef => ref.getKind match {
                        case METHOD =>
                            highlights.put(hiRange, ColoringAttributes.FIELD_SET)
                        case PARAMETER =>
                            highlights.put(hiRange, ColoringAttributes.PARAMETER_SET)
                        case _ =>
                    }
            }
        }
    }
}

In ErlangSemanticeAnalyzer, what you need to do is traversing all AstDfn and AstRef instances, and give them a proper ColoringAttributes set. I marked all functions as METHOD_SET(mod-method), attributes as STATIC_SET(mod-static), and function call names as FIELD_SET(mod-field), parameters as PARAMETER_SET(mod-parameter), the names in parenthesis are the corresponding names that in fontColors.xml setting file, you can define the highlighting effects in this xml file.

Again, don't forget to register it in ErlangLanguage.scala:

override def getSemanticAnalyzer = new ErlangSemanticAnalyzer

Run it, and I got highlighted source code as:

nn

The function declaration names are underline, parameters are bold, function calls are bold too.

I'll gather even more detailed semantic information later, so I can identity the unused variable definitions and var references without definitions etc.

Erlang Plugin for NetBeans in Scala#5: Structure Analyzer

During the weekend, I've done some preliminary error recover work for Erlang's rats! definition. Now I'll go on some features based on analysis on AST tree. As the simplest step, we'll visit/analyze AST tree to get the structure information, use them for Navigator window and code folding.

First, we need to record the Structure information in some way. Each language has different AST tree and may be generated by varies tools, the AST element should be wrapped in some more generic classes to get integrated into CSL framework. There is an interface "org.netbeans.modules.csl.api.ElementKind", which is used for this purpose. But it's not enough, we need some facilities to not only wrap AST element, and also help us to identify the element as a definition or reference, in which visible scope etc.

I wrote a couple of these facilities when I wrote Erlang/Fortress/Scala plug-ins, these facilities can be repeatedly used for other languages by minor modifying. They are AstItem, AstDfn, AstRef, AstScope, AstRootScope. For Erlang plugin, you can find them under director: erlang.editor/src/org/netbeans/modules/erlang/editor/ast

AstDfn.scala is used to store a definition, such as definitions of class, module, method, variable, attribute etc. AstRef.scala is used to store reference that refers to definition, for example, the usages of a class, a variable, a method call etc.

All these AST items are stored in instances of AstScope.scala, which, is a container to identify the visible scope of references/definitions and store its sub-scopes. There are functions in AstScope to help to identify a variable's visible scope, find the definition of a reference, find references/occurrences of a definition etc.

There should be some lexer level utilities too, to help you get the corresponding tokens when you visit AST tree. It's LexUtil.scala, which will be also heavy used for code-folding, indentation ... features.

With above facilities, we can visit an AST tree now, I'll do the simplest task first: get all functions/attribues name and bounds tokens.

AstNodeVisitor.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 * 
 * Copyright 2008 Sun Microsystems, Inc. All rights reserved.
 * 
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 * 
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 * 
 * Contributor(s):
 * 
 * Portions Copyrighted 2009 Sun Microsystems, Inc.
 */
package org.netbeans.modules.erlang.editor.node

import org.netbeans.api.lexer.{Token, TokenId, TokenHierarchy, TokenSequence}
import org.netbeans.modules.csl.api.ElementKind

import org.netbeans.modules.erlang.editor.ast.{AstDfn, AstItem, AstRef, AstRootScope, AstScope, AstVisitor}
import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId._
import org.netbeans.modules.erlang.editor.lexer.{ErlangTokenId, LexUtil}
import org.openide.filesystems.FileObject

import scala.collection.mutable.ArrayBuffer

import xtc.tree.GNode
import xtc.tree.Node
import xtc.util.Pair

/**
 *
 * @author Caoyuan Deng
 */
class AstNodeVisitor(rootNode:Node, th:TokenHierarchy[_], fo:FileObject) extends AstVisitor(rootNode, th) {

    def visitS(that:GNode) = {
        val formNodes = that.getList(0).iterator
        while(formNodes.hasNext) {
            visitForm(formNodes.next)
        }
    }

    def visitForm(that:GNode) = {
        enter(that)

        val scope = new AstScope(boundsTokens(that))
        rootScope.addScope(scope)

        scopes.push(scope)
        visitNodeOnly(that.getGeneric(0))
        
        exit(that)
        scopes.pop
    }


    def visitAttribute(that:GNode) = {
        that.get(0) match {
            case atomId:GNode =>
                val attr = new AstDfn(that, idToken(idNode(atomId)), scopes.top, ElementKind.ATTRIBUTE, fo)
                rootScope.addDfn(attr)
        }
    }

    def visitFunction(that:GNode) = {
        visitFunctionClauses(that.getGeneric(0))
    }

    def visitFunctionClauses(that:GNode) = {
        val fstClauseNode = that.getGeneric(0)
        visitFunctionClause(fstClauseNode)
    }

    def visitFunctionClause(that:GNode) = {
        val id = idNode(that.getGeneric(0))
        val fun = new AstDfn(that, idToken(id), scopes.top, ElementKind.METHOD, fo)
        rootScope.addDfn(fun)
    }

    def visitRule(that:GNode) = {

    }
}

As you can see, I just visit function/attribute related AST nodes, and gather all function/attribute declarations (or, definitions). My AST tree is generated by rats!, so the code looks like above. If you are using other parsers, I assume you know of course how to travel it.

AstNodeVisitor extends AstVisitor, I wrote some helper methods in AstVisitor.scala, for example, getting the bounds tokens for a definition/scope.

Now, you need to put the AST visiting task to ErlangParser.scala, so, it will be called when parsing finished, and you'll get an AST tree. The code is something like:

    private def analyze(context:Context) :Unit = {
        val doc = LexUtil.document(context.snapshot, false)

        // * we need TokenHierarchy to do anaylzing task
        for (root <- context.root;
             th <- LexUtil.tokenHierarchy(context.snapshot)) {
            // * Due to Token hierarchy will be used in analyzing, should do it in an Read-lock atomic task
            for (x <- doc) {x.readLock}
            try {
                val visitor = new AstNodeVisitor(root, th, context.fo)
                visitor.visit(root)
                context.rootScope = Some(visitor.rootScope)
            } catch {
                case ex:Throwable => ex.printStackTrace
            } finally {
                for (x <- doc) {x.readUnlock}
            }
        }
    }

The rootScope will be carried in ErlangParserResult.scala.

With the visitor's rootScope, we can implement the navigator window and code folding now. What you need is to implement an org.netbeans.modules.csl.api.StructureScanner. My implementation is: ErlangStructureAnalyzer.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 *
 * Contributor(s):
 *
 * Portions Copyrighted 2009 Sun Microsystems, Inc.
 */
package org.netbeans.modules.erlang.editor;

import _root_.java.util.{ArrayList,Collections,HashMap,List,Map,Set,Stack}
import javax.swing.ImageIcon;
import javax.swing.text.{BadLocationException,Document}
import org.netbeans.api.lexer.{Token,TokenId,TokenHierarchy,TokenSequence}
import org.netbeans.editor.{BaseDocument,Utilities}
import org.netbeans.modules.csl.api.{ElementHandle,ElementKind,HtmlFormatter,Modifier,OffsetRange,StructureItem,StructureScanner}
import org.netbeans.modules.csl.api.StructureScanner._
import org.netbeans.modules.csl.spi.ParserResult
import org.netbeans.modules.erlang.editor.ast.{AstDfn,AstRootScope,AstScope}
import org.netbeans.modules.erlang.editor.lexer.{ErlangTokenId,LexUtil}
import org.openide.util.Exceptions

import scala.collection.mutable.ArrayBuffer

/**
 *
 * @author Caoyuan Deng
 */
class ErlangStructureAnalyzer extends StructureScanner {

    override
    def getConfiguration :Configuration = null

    override
    def scan(result:ParserResult) :List[StructureItem] = result match {
        case null => Collections.emptyList[StructureItem]
        case pResult:ErlangParserResult => 
            var items = Collections.emptyList[StructureItem]
            for (rootScope <- pResult.rootScope) {
                items = new ArrayList[StructureItem](rootScope.dfns.size)
                scanTopForms(rootScope, items, pResult)
            }
            items
    }

    private def scanTopForms(scope:AstScope, items:List[StructureItem], pResult:ErlangParserResult) :Unit = {
        for (dfn <- scope.dfns) {
            dfn.getKind match {
                case ElementKind.ATTRIBUTE | ElementKind.METHOD => items.add(new ErlangStructureItem(dfn, pResult))
                case _ =>
            }
            scanTopForms(dfn.bindingScope, items, pResult)
        }
    }

    override
    def folds(result:ParserResult) :Map[String, List[OffsetRange]] = result match {
        case null => Collections.emptyMap[String, List[OffsetRange]]
        case pResult:ErlangParserResult =>
            var folds = Collections.emptyMap[String, List[OffsetRange]]
            for (rootScope <- pResult.rootScope;
                 doc <- LexUtil.document(pResult, true);
                 th <- LexUtil.tokenHierarchy(pResult);
                 ts <- LexUtil.tokenSequence(th, 1)
            ) {
                folds = new HashMap[String, List[OffsetRange]]
                val codefolds = new ArrayList[OffsetRange]
                folds.put("codeblocks", codefolds); // NOI18N

                // * Read-lock due to Token hierarchy use
                doc.readLock

                addCodeFolds(pResult, doc, rootScope.dfns, codefolds)

                var lineCommentStart = 0
                var lineCommentEnd = 0
                var startLineCommentSet = false

                val comments = new Stack[Array[Integer]]
                val blocks = new Stack[Integer]

                while (ts.isValid && ts.moveNext) {
                    val token = ts.token
                    token.id match {
                        case ErlangTokenId.LineComment =>
                            val offset = ts.offset
                            if (!startLineCommentSet) {
                                lineCommentStart = offset
                                startLineCommentSet = true
                            }
                            lineCommentEnd = offset

                        case ErlangTokenId.Case | ErlangTokenId.If | ErlangTokenId.Try | ErlangTokenId.Receive =>
                            val blockStart = ts.offset
                            blocks.push(blockStart)

                            startLineCommentSet = false
                        
                        case ErlangTokenId.End if !blocks.empty =>
                            val blockStart = blocks.pop.asInstanceOf[Int]
                            val blockRange = new OffsetRange(blockStart, ts.offset + token.length)
                            codefolds.add(blockRange)

                            startLineCommentSet = false
                        case _ =>
                            startLineCommentSet = false
                    }
                }

                doc.readUnlock

                try {
                    /** @see GsfFoldManager#addTree() for suitable fold names. */
                    lineCommentEnd = Utilities.getRowEnd(doc, lineCommentEnd)

                    if (Utilities.getRowCount(doc, lineCommentStart, lineCommentEnd) > 1) {
                        val lineCommentsFolds = new ArrayList[OffsetRange];
                        val range = new OffsetRange(lineCommentStart, lineCommentEnd)
                        lineCommentsFolds.add(range)
                        folds.put("comments", lineCommentsFolds) // NOI18N
                    }
                } catch {
                    case ex:BadLocationException => Exceptions.printStackTrace(ex)
                }
            }

            folds
    }

    @throws(classOf[BadLocationException])
    private def addCodeFolds(pResult:ErlangParserResult, doc:BaseDocument, defs:ArrayBuffer[AstDfn], codeblocks:List[OffsetRange]) :Unit = {
        import ElementKind._
       
        for (dfn <- defs) {
            val kind = dfn.getKind
            kind match {
                case FIELD | METHOD | CONSTRUCTOR | CLASS | MODULE | ATTRIBUTE =>
                    var range = dfn.getOffsetRange(pResult)
                    var start = range.getStart
                    // * start the fold at the end of the line behind last non-whitespace, should add 1 to start after "->"
                    start = Utilities.getRowLastNonWhite(doc, start) + 1
                    val end = range.getEnd
                    if (start != -1 && end != -1 && start < end && end <= doc.getLength) {
                        range = new OffsetRange(start, end)
                        codeblocks.add(range)
                    }
                case _ =>
            }
    
            val children = dfn.bindingScope.dfns
            addCodeFolds(pResult, doc, children, codeblocks)
        }
    }

    private class ErlangStructureItem(val dfn:AstDfn, pResult:ParserResult) extends StructureItem {
        import ElementKind._

        override
        def getName :String = dfn.getName

        override
        def getSortText :String = getName

        override
        def getHtml(formatter:HtmlFormatter) :String = {
            dfn.htmlFormat(formatter)
            formatter.getText
        }

        override
        def getElementHandle :ElementHandle = dfn

        override
        def getKind :ElementKind = dfn.getKind
        
        override
        def getModifiers :Set[Modifier] = dfn.getModifiers

        override
        def isLeaf :Boolean = dfn.getKind match {
            case MODULE | CLASS => false
            case CONSTRUCTOR | METHOD | FIELD | VARIABLE | OTHER | PARAMETER | ATTRIBUTE => true
            case _ => true
        }

        override
        def getNestedItems : List[StructureItem] = {
            val nested = dfn.bindingScope.dfns
            if (nested.size > 0) {
                val children = new ArrayList[StructureItem](nested.size)

                for (child <- nested) {
                    child.kind match {
                        case PARAMETER | VARIABLE | OTHER =>
                        case _ => children.add(new ErlangStructureItem(child, pResult))
                    }
                }

                children
            } else Collections.emptyList[StructureItem]
        }

        override
        def getPosition :Long = {
            try {
                LexUtil.tokenHierarchy(pResult) match {
                    case None => 0
                    case Some(th) => dfn.boundsOffset(th)
                }
            } catch {case ex:Exception => 0}
        }

        override
        def getEndPosition :Long = {
            try {
                LexUtil.tokenHierarchy(pResult) match {
                    case None => 0
                    case Some(th) => dfn.boundsEndOffset(th)
                }
            } catch {case ex:Exception => 0}
        }

        override
        def equals(o:Any) :Boolean = o match {
            case null => false
            case x:ErlangStructureItem if dfn.getKind == x.dfn.getKind && getName.equals(x.getName) => true
            case _ => false
        }

        override
        def hashCode :Int = {
            var hash = 7
            hash = (29 * hash) + (if (getName != null) getName.hashCode else 0)
            hash = (29 * hash) + (if (dfn.getKind != null) dfn.getKind.hashCode else 0)
            hash
        }

        override
        def toString = getName

        override
        def getCustomIcon :ImageIcon = null
    }
}

Which just travels the rootScope, fetches AstDfn's instances and wrap them to ErlangStructureItem.

The last step is register this structure analyzer to ErlangLanguage.Scala as usual:

    override
    def hasStructureScanner = true

    override
    def getStructureScanner = new ErlangStructureAnalyzer

For structure analyzer, you need also to register it in layer.xml:

    <folder name="CslPlugins">
        <folder name="text">
            <folder name="x-erlang">
                <file name="language.instance">
                    <attr name="instanceClass" stringvalue="org.netbeans.modules.erlang.editor.ErlangLanguage"/>
                </file>
                <file name="structure.instance">
                    <attr name="instanceClass" stringvalue="org.netbeans.modules.erlang.editor.ErlangStructureAnalyzer"/>
                </file>
            </folder>
        </folder>
    </folder>

Build and run, now open a .erl file, you got:

Click on the picture to enlarge it

nn

The functions/attributes are shown in the left-side navigator window now, there are also some code-folding marks. Did you notice my preliminaty error-recover work? :-)

Erlang Plugin for NetBeans in Scala#4: Minimal Parser Intergation

With lexer integrated, you've got all tokens of source file. You can code for pretty formatting, indentation, auto pair inserting, pair matching based on tokens now. But I'll try to integrate an Erlang parser first to get the basic infrastructure ready.

I can choose to use Erlang's native compiler, than via jInterface, Scala can rpc call Erlang runtime to parse source file and return an AST tree. But, Erlang's library function for parsing is not good on error-recover currently, so I choose to write an Erlang parser in Rats!. Rats! generated parser is bad on error messages and error-recover too, but I know how to improve it.

Migrating a NetBeans Schliemann based grammar definition to Rats! is almost straightforward. For me, as I'm already familiar with Rats!, it's an one day work. The preliminary Rats! definition for Erlang can be found at ParserErlang.rats. Then, I rewrote a new ParserErlang.rats according to latest Erlang spec in another half day, without the LL(k) limitation, the grammar rules now keep as close as the original spec.

To get it be used to generate a ParserErlang.java, I added corresponding ant target to build.xml, which looks like:

    <target name="rats" depends="init" description="Scanner">
        <echo message="Rebuilding token scanner... ${rats.package.dir}"/>
        <java fork="yes"
             dir="${src.dir}/${rats.package.dir}"
             classname="xtc.parser.Rats"
             classpath="${rats.jar}">
            <arg value="-in"/>
            <arg value="${src.dir}"/>
            <arg value="${rats.lexer.file}"/>
        </java>

        <echo message="Rebuilding grammar parser... ${rats.package.dir}"/>
        <java fork="yes"
             dir="${src.dir}/${rats.package.dir}"
             classname="xtc.parser.Rats"
             classpath="${rats.jar}">
            <arg value="-in"/>
            <arg value="${src.dir}"/>
            <arg value="${rats.parser.file}"/>
        </java>
    </target>

And rats related properties, such as "rats.lexer.file" and "rats.parser.file" are defined in nbproject/project.properties as:

rats.jar=${cluster}/modules/xtc.jar
rats.package.dir=org/netbeans/modules/erlang/editor/rats
rats.lexer.file=LexerErlang.rats
rats.parser.file=ParserErlang.rats

Running target "rats" will generate LexerErlang.java and ParserErlang.java, the first one is that we've used in ErlangLexer.scala, the later one is to be integrated into NetBeans' Parsing API as the parser.

Now, you should extend two Parsing API abstract classes, org.netbeans.modules.csl.spi.ParserResult and org.netbeans.modules.parsing.spi.Parser. The first one will carry result's AST Node and syntax errors that parser detected, the errors will be highlighted automatically in Editor. The second one is the bridge between NetBeans parsing task and your real parser (ParserErlang.java here)

There are tricks to do some error-recover in these two classes, by adding a "." or "end" or "}" etc to the buffered source chars when a syntax error occurred, it's called "sanitize" the source to recover the error. NetBeans' Ruby, JavaScripts supporting have some good examples of this trick. For my case, I will do error recover in Rats! definition later, so I do not use this trick currently, but I leave some "sanitize" related code there.

First, it's an ErlangParserResult.scala which extended ParserResult. The code is simple at the first phase.

ErlangParserResult.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor

import _root_.java.util.{Collections, ArrayList, List}
import org.netbeans.api.lexer.{TokenHierarchy, TokenId}
import org.netbeans.modules.csl.api.Error
import org.netbeans.modules.csl.api.OffsetRange
import org.netbeans.modules.csl.spi.ParserResult
import org.netbeans.modules.parsing.api.Snapshot
import org.netbeans.modules.erlang.editor.rats.ParserErlang
import xtc.tree.{GNode}

/**
 *
 * @author Caoyuan Deng
 */
class ErlangParserResult(parser:ErlangParser,
                         snapshot:Snapshot,
                         val rootNode:GNode,
                         val th:TokenHierarchy[_]) extends ParserResult(snapshot) {

    override
    protected def invalidate :Unit = {
        // XXX: what exactly should we do here?
    }

    override
    def getDiagnostics :List[Error] = _errors

    private var _errors = Collections.emptyList[Error]
    
    def errors = _errors
    def errors_=(errors:List[Error]) = {
        this._errors = new ArrayList[Error](errors)
    }

    var source :String = _
    
    /**
     * Return whether the source code for the parse result was "cleaned"
     * or "sanitized" (modified to reduce chance of parser errors) or not.
     * This method returns OffsetRange.NONE if the source was not sanitized,
     * otherwise returns the actual sanitized range.
     */
    var sanitizedRange = OffsetRange.NONE
    var sanitizedContents :String = _
    var sanitized :Sanitize = NONE

    var isCommentsAdded :Boolean = false
    
    /**
     * Set the range of source that was sanitized, if any.
     */
    def setSanitized(sanitized:Sanitize, sanitizedRange:OffsetRange, sanitizedContents:String) :Unit = {
        this.sanitized = sanitized
        this.sanitizedRange = sanitizedRange
        this.sanitizedContents = sanitizedContents
    }

    override
    def toString = {
        "ErlangParseResult(file=" + snapshot.getSource.getFileObject + ",rootnode=" + rootNode + ")"
    }
}

Then, the ErlangParser.scala which extended Parser:

ErlangParser.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor


import _root_.java.io.{IOException, StringReader}
import _root_.java.util.ArrayList
import _root_.java.util.Collection
import _root_.java.util.List
import _root_.java.util.ListIterator
import _root_.javax.swing.event.ChangeListener
import _root_.javax.swing.text.BadLocationException

import org.netbeans.modules.csl.api.ElementHandle
import org.netbeans.modules.csl.api.Error
import org.netbeans.modules.csl.api.OffsetRange
import org.netbeans.modules.csl.api.Severity
import org.netbeans.modules.csl.spi.DefaultError
import org.netbeans.modules.parsing.api.Snapshot
import org.netbeans.modules.parsing.api.Task
import org.netbeans.modules.parsing.spi.ParseException
import org.netbeans.modules.csl.api.EditHistory
import org.netbeans.modules.csl.spi.GsfUtilities
import org.netbeans.modules.csl.spi.ParserResult
import org.netbeans.modules.parsing.api.Source
import org.netbeans.modules.parsing.spi.Parser
import org.netbeans.modules.parsing.spi.Parser.Result
import org.netbeans.modules.parsing.spi.ParserFactory
import org.netbeans.modules.parsing.spi.SourceModificationEvent
import org.openide.filesystems.FileObject
import org.openide.util.Exceptions

import org.netbeans.api.editor.EditorRegistry
import org.netbeans.api.lexer.{TokenHierarchy, TokenId}
import org.netbeans.editor.BaseDocument
import org.netbeans.modules.editor.NbEditorUtilities

import xtc.parser.{ParseError, SemanticValue}
import xtc.tree.{GNode, Location}

import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId
import org.netbeans.modules.erlang.editor.rats.ParserErlang

/**
 *
 * @author Caoyuan Deng
 */
class ErlangParser extends Parser {

    private var lastResult :ErlangParserResult = _

    @throws(classOf[ParseException])
    override
    def parse(snapshot:Snapshot, task:Task, event:SourceModificationEvent) :Unit = {
        val context = new Context(snapshot, event)
        lastResult = parseBuffer(context, NONE)
        lastResult.errors = context.errors
    }

    @throws(classOf[ParseException])
    override
    def getResult(task:Task) :Result = {
        assert(lastResult != null, "getResult() called prior parse()") //NOI18N
        lastResult
    }

    override
    def cancel :Unit = {}

    override
    def addChangeListener(changeListener:ChangeListener) :Unit = {
        // no-op, we don't support state changes
    }

    override
    def removeChangeListener(changeListener:ChangeListener) :Unit = {
        // no-op, we don't support state changes
    }

    private def lexToAst(source:Snapshot, offset:Int) :Int = source match {
        case null => offset
        case _ => source.getEmbeddedOffset(offset)
    }

    private def astToLex(source:Snapshot, offset:Int) :Int = source match {
        case null => offset
        case _ => source.getOriginalOffset(offset)
    }

    private def sanitizeSource(context:Context, sanitizing:Sanitize) :Boolean = {
        false
    }

    private def sanitize(context:Context, sanitizing:Sanitize) :ErlangParserResult = {
        sanitizing match {
            case NEVER =>
                createParseResult(context)
            case NONE =>
                createParseResult(context)
            case _ =>
                // we are out of trick, just return as it
                createParseResult(context)
        }
    }

    protected def notifyError(context:Context, message:String, sourceName:String,
                              start:Int, lineSource:String, end:Int,
                              sanitizing:Sanitize, severity:Severity,
                              key:String, params:Object) :Unit = {

        val error = new DefaultError(key, message, null, context.fo, start, end, severity)

        params match {
            case null =>
            case x:Array[Object] => error.setParameters(x)
            case _ => error.setParameters(Array(params))
        }

        context.notifyError(error)

        if (sanitizing == NONE) {
            context.errorOffset = start
        }
    }

    protected def parseBuffer(context:Context, sanitizing:Sanitize) :ErlangParserResult = {
        var sanitizedSource = false
        var source = context.source

        sanitizing match {
            case NONE | NEVER =>
            case _ =>
                val ok = sanitizeSource(context, sanitizing)
                if (ok) {
                    assert(context.sanitizedSource != null)
                    sanitizedSource = true
                    source = context.sanitizedSource
                } else {
                    // Try next trick
                    return sanitize(context, sanitizing)
                }
        }

        if (sanitizing == NONE) {
            context.errorOffset = -1
        }

        val parser = createParser(context)

        val ignoreErrors = sanitizedSource
        var root :GNode = null
        try {
            var error :ParseError = null
            val r = parser.pS(0)
            if (r.hasValue) {
                val v = r.asInstanceOf[SemanticValue]
                root = v.value.asInstanceOf[GNode]
            } else {
                error = r.parseError
            }

            if (error != null && !ignoreErrors) {
                var start = 0
                if (error.index != -1) {
                    start = error.index
                }
                notifyError(context, error.msg, "Syntax error",
                            start, "", start,
                            sanitizing, Severity.ERROR,
                            "SYNTAX_ERROR", Array(error))

                System.err.println(error.msg)
            }

        } catch {
            case e:IOException => e.printStackTrace
            case e:IllegalArgumentException =>
                // An internal exception thrown by parser, just catch it and notify
                notifyError(context, e.getMessage, "",
                            0, "", 0,
                            sanitizing, Severity.ERROR,
                            "SYNTAX_ERROR", Array(e))
        }

        if (root != null) {
            context.sanitized = sanitizing
            context.root = root
            val r = createParseResult(context)
            r.setSanitized(context.sanitized, context.sanitizedRange, context.sanitizedContents)
            r.source = source
            r
        } else {
            sanitize(context, sanitizing)
        }
    }

    protected def createParser(context:Context) :ParserErlang = {
        val in = new StringReader(context.source)
        val fileName = if (context.fo != null) context.fo.getNameExt else ""

        val parser = new ParserErlang(in, fileName)
        context.parser = parser

        parser
    }

    private def createParseResult(context:Context) :ErlangParserResult = {
        new ErlangParserResult(this, context.snapshot, context.root, context.th)
    }

    /** Parsing context */
    class Context(val snapshot:Snapshot, event:SourceModificationEvent) {
        val errors :List[Error] = new ArrayList[Error]

        var source :String = ErlangParser.asString(snapshot.getText)
        var caretOffset :Int = GsfUtilities.getLastKnownCaretOffset(snapshot, event)

        var root :GNode = _
        var th :TokenHierarchy[_] = _
        var parser :ParserErlang = _
        var errorOffset :Int = _
        var sanitizedSource :String = _
        var sanitizedRange :OffsetRange = OffsetRange.NONE
        var sanitizedContents :String = _
        var sanitized :Sanitize = NONE

        def notifyError(error:Error) = errors.add(error)

        def fo = snapshot.getSource.getFileObject

        override
        def toString = "ErlangParser.Context(" + fo + ")" // NOI18N

    }
}

object ErlangParser {
    def asString(sequence:CharSequence) :String = sequence match {
        case s:String => s
        case _ => sequence.toString
    }

    def sourceUri(source:Source) :String = source.getFileObject match {
        case null => "fileless" //NOI18N
        case f => f.getNameExt
    }
}

/** Attempts to sanitize the input buffer */
sealed case class Sanitize
/** Only parse the current file accurately, don't try heuristics */
case object NEVER extends Sanitize
/** Perform no sanitization */
case object NONE extends Sanitize

The major tasks of these two classes are:

  • Parsing the source buffer which is passed automatically by the framework when source is modified or just opened, the source text can be got from snapshot.getText. The parsing result used to be an AST tree, you can do semantic/structure analysis on AST tree to get more detailed information later, but as the first step, I just pass/keep AST tree in ErlangParserResult for later usage.
  • Notifying the errors to framework. This can be done by store the parsing errors to ParserResult, and implemented getDiagnostics to return the error.

The last step is register the ErlangParser in ErlangLanguage.scala in one line code:

override def getParser = new ErlangParser

Now, open a .erl file, if there is syntax error, the editor will indicate and highlight it now.

My next step is to improve the Rats! definition, add error recover etc.

Erlang Plugin for NetBeans in Scala#2: Set Environment for Writing NetBeans Module in Scala

I'm going to create new Erlang Editor module in Scala, the module project is under http://hg.netbeans.org/main/contrib/file/0caa5d009839/erlang.editor/ which I just committed in.

The first step is to get Scala source file mixed in Java based NetBeans source/project tree. After define project's manifest,mf, we'll create a special build.xml for this project under erlang.editor directory:

build.xml

<?xml version="1.0" encoding="UTF-8"?>
<project name="contrib/erlang.editor" default="netbeans" basedir=".">
    <import file="../../nbbuild/templates/projectized.xml"/>
    <import file="scala-build.xml"/>

    <!-- special jar target for csl -->
    <target name="jar" depends="init,compile,jar-prep" unless="is.jar.uptodate">
        <taskdef name="csljar" classname="org.netbeans.modules.csl.CslJar" classpath="${nb_all}/csl.api/anttask/build/cslanttask.jar:${nb_all}/nbbuild/nbantext.jar"/>
        <csljar jarfile="${cluster}/${module.jar}" compress="${build.package.compress}" index="${build.package.index}" manifest="${manifest.mf}" stamp="${cluster}/.lastModified">
            <fileset dir="${build.classes.dir}"/>
        </csljar>
    </target>

    <target name="compile" depends="init,projectized-common.compile,scala-compile"/>
    <target name="do-test-build" depends="init,test-init,projectized-common.do-test-build"/>

    <target name="rats" depends="init" description="Scanner">
        <echo message="Rebuilding token scanner... ${rats.package.dir}"/>
        <java fork="yes"
             dir="${src.dir}/${rats.package.dir}"
             classname="xtc.parser.Rats"
             classpath="${rats.jar}">
            <arg value="-in"/>
            <arg value="${src.dir}"/>
            <arg value="${rats.lexer.file}"/>
        </java>
    </target>
    
</project>

This above build.xml imported NetBeans' main template ant xml "../../nbbuild/templates/projectized.xml" and a special scala-build.xml file, which defines scalac and javac task for mixed Scala/Java module for NetBeans. You can ignore the "rats" target if you has no plan to write language's lexer/parser in Rats! parser generator. scala-build.xml looks like:

scala-build.xml

<?xml version="1.0" encoding="UTF-8"?>
<project name="scala-module" default="netbeans" basedir=".">
    <import file="../../nbbuild/templates/projectized.xml"/>
    
    <target name="scala-taskdef" depends="init">
        <echo message="Compiling scala sources via ${scala.library}, ${scala.compiler}"/>
        <taskdef resource="scala/tools/ant/antlib.xml">
            <classpath>
                <pathelement location="${scala.library}"/>
                <pathelement location="${scala.compiler}"/>
            </classpath>
        </taskdef>
    </target>

    <property name="jar-excludes" value="**/*.java,**/*.form,**/package.html,**/doc-files/,**/*.scala"/>
    
    <target name="scala-compile" depends="init,up-to-date,scala-taskdef" unless="is.jar.uptodate">
        <!-- javac's classpath should include scala.library and all these paths of "cp" -->
        <path id="javac.cp">
            <pathelement path="${scala.libs}"/>
            <pathelement path="${module.classpath}"/>
            <pathelement path="${cp.extra}"/>
        </path>
        <!-- scalac will check class dependencies deeply, so we can not rely on public package only which is refed by ${module.classpath} -->
        <path id="scalac.cp">
            <pathelement path="${scala.libs}"/>
            <pathelement path="${module.run.classpath}"/>
            <pathelement path="${cp.extra}"/>
        </path>
        <mkdir dir="${build.classes.dir}"/>
        <depend srcdir="${src.dir}" destdir="${build.classes.dir}" cache="build/depcache">
            <classpath refid="scalac.cp"/>
        </depend>
        <!-- scalac -->
        <scalac srcdir="${src.dir}" destdir="${build.classes.dir}" encoding="UTF-8" target="jvm-${javac.target}">
            <classpath refid="scalac.cp"/>
        </scalac>
        <!-- javac -->
        <nb-javac srcdir="${src.dir}" destdir="${build.classes.dir}" debug="${build.compiler.debug}" debuglevel="${build.compiler.debuglevel}" encoding="UTF-8"
                deprecation="${build.compiler.deprecation}" optimize="${build.compiler.optimize}" source="${javac.source}" target="${javac.target}" includeantruntime="false">
            <classpath refid="javac.cp"/>
            <compilerarg line="${javac.compilerargs}"/>
            <processorpath refid="processor.cp"/>
        </nb-javac>
        <!-- Sanity check: -->
        <pathconvert pathsep=":" property="class.files.in.src">
            <path>
                <fileset dir="${src.dir}">
                    <include name="**/*.class"/>
                </fileset>
            </path>
        </pathconvert>
        <fail>
            <condition>
                <not>
                    <equals arg1="${class.files.in.src}" arg2=""/>
                </not>
            </condition>
            You have stray *.class files in ${src.dir} which you must remove.
            Probably you failed to clean your sources before updating them.
        </fail>
        <!-- OK, continue: -->
        <copy todir="${build.classes.dir}">
            <fileset dir="${src.dir}" excludes="${jar-excludes}"/>
        </copy>
    </target>

    <target name="do-test-build" depends="projectized-common.do-test-build">
        <scalac srcdir="${test.unit.src.dir}" destdir="${build.test.unit.classes.dir}" excludes="${test.excludes}"
               encoding="UTF-8">
            <classpath refid="test.unit.cp"/>
        </scalac>
    </target>
</project>

You need also to set the project's module dependencies on:

  • org.netbeans.libs.scala
  • org.netbeans.modules.csl.api
  • org.netbeans.modules.lexer
  • org.netbeans.modules.parsing.api
  • org.openide.filesystems
etc, where org.netbeans.libs.scala is scala's run time module, which is at contrib/libs.scala

And, the nbproject/project.properties which defines some important project building properties, such as:

javac.compilerargs=-Xlint:unchecked
javac.source=1.5
nbm.homepage=http://wiki.netbeans.org/Erlang

scala.library=${cluster}/modules/ext/scala-library-2.7.3.jar
scala.compiler=${cluster}/modules/ext/scala-compiler-2.7.3.jar
scala.libs=\
   ${scala.library}:\
   ${scala.compiler}

You can write NetBeans' module in Scala now.

Erlang Plugin for NetBeans in Scala#3: Minimal Lexer Intergation

>>> Updated Feb 8:

Code fixed to avoid an infinite cycled scanning when source file size > 16k

===

The minim supporting is to integrate a language lexer to NetBeans's language support framework. As NetBeans 7.0, there is a new effort for common language supporting, which is called CSL (Common Scripting Language). Don't be confused by this module's name, it not only for scripting language, actually, Java support has been migrated to CSL in 7.0. There are discussions on a better name. CSL is forked and created on GSF (Generic Scripting Framework) as a GSF's new variant that is based on new Parsing & Indexing API.

Since I'm going to write an Erlang lexer in Rats! generator parser, I need to add dependency on rats run-time libs first, rats run-time module is under contrib/xtc, which I patched to support end position of each production.

Then, you have to tell the project where to find the rats! libs, this can be done by adding following properties to nbproject/project.properties, now this nbproject/project.properties looks like:

nbproject/project.properties

javac.compilerargs=-Xlint:unchecked
javac.source=1.5
nbm.homepage=http://wiki.netbeans.org/Erlang

scala.library=${cluster}/modules/ext/scala-library-2.7.3.jar
scala.compiler=${cluster}/modules/ext/scala-compiler-2.7.3.jar
scala.libs=\
   ${scala.library}:\
   ${scala.compiler}

rats.jar=${cluster}/modules/xtc.jar
rats.package.dir=org/netbeans/modules/erlang/editor/rats
rats.lexer.file=LexerErlang.rats

All Rats! definitions of Erlang token can be found at http://hg.netbeans.org/main/contrib/file/tip/erlang.editor/src/org/netbeans/modules/erlang/editor/rats/. Don't ask me how to write Rats! rules for languages, you should get these information from Rats! web site. Or, you can integrate other types of lexer generated by other lexer generator, the examples can be found in NetBeans' other languages supporting modules.

The Erlang's lexer will be generated via "rats.lexer.file=LexerErlang.rats", which is the entry point of all defined rules for Erlang tokens. Run "rats" target will generate a LexerErlang.java file which is the lexer class that will be used to create Erlang tokens from Erlang source files.

Now, we should integrate this lexer class to NetBeans' lexer engine, this is done by two Scala files:

ErlangLexer.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2008 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor.lexer

import _root_.java.io.IOException
import _root_.java.io.Reader
import org.netbeans.api.lexer.{Token, TokenId}
import org.netbeans.modules.erlang.editor.rats.LexerErlang
import org.netbeans.spi.lexer.Lexer
import org.netbeans.spi.lexer.LexerInput
import org.netbeans.spi.lexer.LexerRestartInfo
import org.netbeans.spi.lexer.TokenFactory
import xtc.parser.Result
import xtc.tree.GNode
import xtc.util.Pair

import scala.collection.mutable.ArrayBuffer

import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId._

/**
 *
 * @author Caoyuan Deng
 */
object ErlangLexer {
    /** @Note:
     * ErlangLexer class is not Reentrant safe, it seems when source size is large than 16 * 1024,
     * there will be more than one input are used, which causes the offset states, such as readed 
     * token length, offset etc in these inputs conflict?. Anyway it's safe to create a new one always.
     */
    def create(info:LexerRestartInfo[TokenId]) = new ErlangLexer(info)
}

class ErlangLexer(info:LexerRestartInfo[TokenId]) extends Lexer[TokenId] {
    /** @Note:
     * it seems input at this time is empty, so we can not do scanning here.
     * input will be filled in chars when call nextToken
     */

    var input :LexerInput = info.input
    var tokenFactory :TokenFactory[TokenId] = info.tokenFactory
    var lexerInputReader :LexerInputReader = new LexerInputReader(input)
    
    val tokenStream = new ArrayBuffer[TokenInfo]
    // * tokenStream.elements always return a new iterator, which point the first
    // * item, so we should have a global one.
    var tokenStreamItr :Iterator[TokenInfo]  = tokenStream.elements
    var lookahead :Int = 0

    override
    def release = {}

    override
    def state :Object = null

    override
    def nextToken :Token[TokenId] = {
        // * In case of embedded tokens, there may be tokens that had been scanned
        // * but not taken yet, check first
        if (!tokenStreamItr.hasNext) {
            tokenStream.clear
            scanTokens
            tokenStreamItr = tokenStream.elements

            /**
             * @Bug of LexerInput.backup(int) ?
             * backup(0) will cause input.readLength() increase 1
             */
            lookahead = input.readLength
            if (lookahead > 0) {
                // * backup all, we will re-read from begin to create token at following step
                input.backup(lookahead)
            } else {
                return null
            }
        }

        if (tokenStreamItr.hasNext) {
            val tokenInfo = tokenStreamItr.next

            if (tokenInfo.length == 0) {
                // * EOF
                return null
            }

            // * read token's chars according to tokenInfo.length
            var i = 0
            while (i < tokenInfo.length) {
                input.read
                i += 1
            }

            // * see if needs to lookahead, if true, perform it
            lookahead -= tokenInfo.length
            // * to cheat incremently lexer, we needs to lookahead one more char when
            // * tokenStream.size() > 1 (batched tokens that are not context free),
            // * so, when modification happens extractly behind latest token, will
            // * force lexer relexer from the 1st token of tokenStream
            val lookahead1 = if (tokenStream.size > 1) lookahead + 1 else lookahead
            if (lookahead1 > 0) {
                var i = 0
                while (i < lookahead1) {
                    input.read
                    i += 1
                }
                input.backup(lookahead1)
            }

            val tokenLength = input.readLength
            createToken(tokenInfo.id, tokenLength)
        } else {
            assert(false, "unrecognized input" + input.read)
            null
        }
    }

    def createToken(id:TokenId, length:Int) :Token[TokenId] = id.asInstanceOf[ErlangTokenId].fixedText match {
        case null => tokenFactory.createToken(id, length)
        case fixedText => tokenFactory.getFlyweightToken(id, fixedText)
    }

    def scanTokens :Result = {
        /**
         * We cannot keep an instance scope lexer, since lexer (sub-class of ParserBase)
         * has internal states which keep the read-in chars, index and others, it really
         * difficult to handle.
         */
        val scanner = new LexerErlang(lexerInputReader, "")
        try {
            // * just scan from position 0, incrmental lexer engine will handle start char in lexerInputReader
            val r = scanner.pToken(0)
            if (r.hasValue) {
                val node = r.semanticValue.asInstanceOf[GNode]
                flattenToTokenStream(node)
                r
            } else {
                System.err.println(r.parseError.msg)
                null
            }
        } catch {
            case e:Exception =>
                System.err.println(e.getMessage)
                null
        }
    }

    def flattenToTokenStream(node:GNode) :Unit = {
        val l = node.size
        if (l == 0) {
            /** @Note:
             * When node.size == 0, it's a void node. This should be limited to
             * EOF when you define lexical rats.
             *
             * And in Rats!, EOF is !_, the input.readLength() will return 0
             */      
            assert(input.readLength == 0,
                   "This generic node: " + node.getName +
                   " is a void node, this should happen only on EOF. Check you rats file.")

            val tokenInfo = new TokenInfo(0, null)
            tokenStream += tokenInfo
            return
        }
        
        var i = 0
        while (i < l) {
            node.get(i) match {
                case null =>
                    // * child may be null
                case child:GNode =>
                    flattenToTokenStream(child)
                case child:Pair[_] =>
                    assert(false, "Pair:" + child + " to be process, do you add 'flatten' option on grammar file?")
                case child:String =>
                    val length = child.length
                    val id = ErlangTokenId.valueOf(node.getName) match {
                        case None => ErlangTokenId.IGNORED
                        case Some(v) => v.asInstanceOf[TokenId]
                    }
          
                    val tokenInfo = new TokenInfo(length, id)
                    tokenStream += tokenInfo
                case child =>
                    println("To be process: " + child)
            }
            i += 1
        }
    }

    /**
     * Hacking for xtc.parser.ParserBase of Rats! which use java.io.Reader
     * as the chars input, but uses only {@link java.io.Reader#read()} of all methods in
     * {@link xtc.parser.ParserBase#character(int)}
     */
    class LexerInputReader(input:LexerInput) extends Reader {
        override
        def read :Int = input.read match {
            case LexerInput.EOF => -1
            case c => c
        }

        override
        def read(cbuf:Array[Char], off:Int, len:Int) :Int = {
            throw new UnsupportedOperationException("Not supported yet.")
        }

        override
        def close = {}
    }

    class TokenInfo(val length:Int, val id:TokenId) {
        override
        def toString = "(id=" + id + ", length=" + length + ")"
    }
}

ErlangTokenId

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor.lexer

import _root_.java.util.Collection
import _root_.java.util.Collections
import _root_.java.util.HashMap
import _root_.java.util.HashSet
import _root_.java.util.Map
import _root_.java.util.Arrays

import org.netbeans.api.lexer.InputAttributes
import org.netbeans.api.lexer.Language
import org.netbeans.api.lexer.LanguagePath
import org.netbeans.api.lexer.Token
import org.netbeans.api.lexer.TokenId
import org.netbeans.spi.lexer.LanguageEmbedding
import org.netbeans.spi.lexer.LanguageHierarchy
import org.netbeans.spi.lexer.Lexer
import org.netbeans.spi.lexer.LexerRestartInfo

/**
 * 
 * @author Caoyuan Deng
 */
object ErlangTokenId extends Enumeration {
    // Let type of enum's value the same as enum itself
    type ErlangTokenId = V

    // Extends Enumeration.Val to get custom enumeration value
    class V(val name:String, val fixedText:String, val primaryCategory:String) extends Val(name) with TokenId {
        override
        def ordinal = id
    }
    object V {
        def apply(name:String, fixedText:String, primaryCategory:String) = new V(name, fixedText, primaryCategory)
    }
  
    val IGNORED = V("IGNORED", null, "ingore")
    val Error = V("Error", null, "error")

    // --- Spaces and comments
    val Ws = V("Ws", null, "whitespace")
    val Nl = V("Nl", null, "whitespace")
    val LineComment = V("LineComment", null, "comment")
    val CommentTag = V("CommentTag", null, "comment")
    val CommentData = V("CommentData", null, "comment")

    // --- Literals
    val IntegerLiteral = V("IntegerLiteral", null, "number")
    val FloatingPointLiteral = V("FloatingPointLiteral", null, "number")
    val CharacterLiteral = V("CharacterLiteral", null, "char")
    val StringLiteral = V("StringLiteral", null, "string")

    // --- Keywords
    val Andalso = V("Andalso", "andalso", "keyword")
    val After = V("After", "after", "keyword")
    val And = V("And", "and", "keyword")
    val Band = V("Band", "band", "keyword")
    val Begin = V("Begin", "begin", "keyword")
    val Bnot = V("Bnot", "bnot", "keyword")
    val Bor = V("Bor", "bor", "keyword")
    val Bsr = V("Bsr", "bsr", "keyword")
    val Bxor = V("Bxor", "bxor", "keyword")
    val Case = V("Case", "case", "keyword")
    val Catch = V("Catch", "catch", "keyword")
    val Cond = V("Cond", "cond", "keyword")
    val Div = V("Div", "div", "keyword")
    val End = V("End", "end", "keyword")
    val Fun = V("Fun", "fun", "keyword")
    val If = V("If", "if", "keyword")
    val Not = V("Not", "not", "keyword")
    val Of = V("Of", "of", "keyword")
    val Orelse = V("Orelse", "orelse", "keyword")
    val Or = V("Or", "or", "keyword")
    val Query = V("Query", "query", "keyword")
    val Receive = V("Receive", "receive", "keyword")
    val Rem = V("Rem", "rem", "keyword")
    val Try = V("Try", "try", "keyword")
    val Spec = V("Spec", "spec", "keyword")
    val When = V("When", "when", "keyword")
    val Xor = V("Xor", "xor", "keyword")

    // --- Identifiers
    val Macro = V("Macro", null, "identifier")
    val Atom = V("Atom", null, "identifier")
    val Var = V("Var", null, "identifier")
    val Rec = V("Rec", null, "identifier")

    // --- Stop
    val Stop = V("Stop", ".", "separator")

    // --- Symbols
    val LParen = V("LParen", "(", "separator")
    val RParen = V("RParan", ")", "separator")
    val LBrace = V("LBrace", "{", "separator")
    val RBrace = V("RBrace", "}", "separator")
    val LBracket = V("LBracket", "[", "separator")
    val RBracket = V("RBracket", "]", "separator")
    val Comma = V("Comma", ",", "separator")
    val Dot = V("Dot", ".", "separator")
    val Semicolon = V("Semicolon", ";", "separator")
    val DBar = V("DBar", "||", "separator")
    val Bar = V("Bar", "|",  "separator")
    val Question = V("Question", "?","separator")
    val DLt = V("DLt", "<<", "separator")
    val LArrow = V("LArrow", "<-", "separator")
    val Lt = V("Lt", "<", "separator")
    val DGt = V("DGt", ">>", "separator")
    val Ge = V("Ge", ">=", "separator")
    val Gt = V("Gt", ">", "separator")
    val ColonMinus = V("ColonMinus", ":-", "separator")
    val DColon = V("DColon", "::", "separator")
    val Colon = V("Colon", ":", "separator")
    val Hash = V("Hash", "#", "separator")
    val DPlus = V("DPlus", "++", "separator")
    val Plus = V("Plus", "+", "separator")
    val DMinus = V("DMinus", "--", "separator")
    val RArrow = V("RArrow", "->", "separator")
    val Minus = V("Minus", "-", "separator")
    val Star = V("Star", "*", "separator")
    val Ne = V("Ne", "/=", "separator")
    val Slash = V("Slash", "/", "separator")
    val EEq = V("EEq", "=:=", "separator")
    val ENe = V("ENe", "=/=", "separator")
    val DEq = V("DEq", "==", "separator")
    val Le = V("le", "=<", "separator")
    val Eq = V("Eq", "=", "separator")
    val Exclamation = V("Exclamation", "!", "separator")

  
    /**
     * MIME type for Erlang. Don't change this without also consulting the various XML files
     * that cannot reference this value directly.
     */
    val ERLANG_MIME_TYPE = "text/x-erlang"; // NOI18N

    // * should use "val" instead of "def" here to get a singleton language val, which  
    // * will be used to identity the token's language by "==" comparasion by other classes.
    // * Be aware of the init order! to get createTokenIds gathers all TokenIds, should
    // * be put after all token id val definition
    val language = new LanguageHierarchy[TokenId] {
        protected def mimeType = ERLANG_MIME_TYPE

        protected def createTokenIds :Collection[TokenId] = {
            val ids = new HashSet[TokenId]
            elements.foreach{ids add _.asInstanceOf[TokenId]}
            ids
        }
    
        protected def createLexer(info:LexerRestartInfo[TokenId]) :Lexer[TokenId] = ErlangLexer.create(info)

        override
        protected def createTokenCategories :Map[String, Collection[TokenId]] = {
            val cats = new HashMap[String, Collection[TokenId]]
            cats
        }

        override
        protected def embedding(token:Token[TokenId], languagePath:LanguagePath, inputAttributes:InputAttributes) = {
            null // No embedding
        }
    }.language

}

ErlangTokenId implemented org.netbeans.api.lexer.TokenId, and defined all Erlang token's Ids, with each name, id, fixedText and primaryCategory.

ErlangLexer.scala is the bridge between LexerErlang.java and NetBeans' lexer engine. NetBeans' lexer engine is an incremental engine, which will automatically handle the positions when you insert/modify a char, wrap the sanitary buffer in a LexerInput and pass it to lexer. I have carefully degined ErlangLexer.scala to get these benefits, you can use this file for other languages too, if you have understood how I write rats rules for tokens.

Now, you should implemented an org.netbeans.modules.csl.spi.DefaultLanguageConfig, which register all services for your language, such as: CodeCompletionHandler, DeclarationFinder, Formatter, IndexSearcher, InstantRenamer, KeystrokeHandler, OccurrencesFinder, SemanticAnalyzer, StructureScanner etc. For the first step, we only implemented a minim supporting for Erlang, which actually is only a lexer to get Erlang tokens and highlight them. So, our implementation is fairly simple:

ErlangLanguage.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor

import _root_.java.io.File
import _root_.java.util.Collection
import _root_.java.util.Collections
import _root_.java.util.HashMap
import _root_.java.util.Map
import _root_.java.util.Set
import org.netbeans.api.lexer.Language;
import org.netbeans.modules.csl.api.CodeCompletionHandler
import org.netbeans.modules.csl.api.DeclarationFinder
import org.netbeans.modules.csl.api.Formatter
import org.netbeans.modules.csl.api.IndexSearcher
import org.netbeans.modules.csl.api.InstantRenamer
import org.netbeans.modules.csl.api.KeystrokeHandler
import org.netbeans.modules.csl.api.OccurrencesFinder
import org.netbeans.modules.csl.api.SemanticAnalyzer
import org.netbeans.modules.csl.api.StructureScanner
import org.netbeans.modules.csl.spi.DefaultLanguageConfig
import org.netbeans.modules.parsing.spi.Parser
import org.netbeans.modules.parsing.spi.indexing.EmbeddingIndexerFactory
import org.openide.filesystems.FileObject
import org.openide.filesystems.FileUtil
import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId

/*
 * Language/lexing configuration for Erlang
 *
 * @author Caoyuan Deng
 */
class ErlangLanguage extends DefaultLanguageConfig {

  override
  def getLexerLanguage = ErlangTokenId.language
    
  override
  def getDisplayName : String =  "Erlang"
    
  override
  def getPreferredExtension : String = {
    "erl" // NOI18N
  }    
}
where def getLexerLanguage = ErlangTokenId.language is exact the LanguageHierarchy implementation for ErlangTokenId in ErlangTokenId.scala, which will tell the framework about token ids, category, embedding information.

The final step is to register ErlangLanguage and fontColor.xml, erlangResolver.xml etc in layer.xml for color highlights, mime resolver and language icon. All these necessary resource files are under: http://hg.netbeans.org/main/contrib/file/0caa5d009839/erlang.editor/src/org/netbeans/modules/erlang/editor/resources/

layer.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE filesystem PUBLIC "-//NetBeans//DTD Filesystem 1.1//EN" "http://www.netbeans.org/dtds/filesystem-1_1.dtd">
<filesystem>
    <folder name="Services">
        <folder name="MIMEResolver">
            <file name="Erlang.xml" url="erlangResolver.xml">
                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.resources.Bundle"/>
                <attr name="position" intvalue="1005"/>
            </file>
        </folder>
    </folder>

    <folder name="Editors">
      <!-- Reference binding color themes are under module: main/defaults/src/org/netbeans/modules/defaults -->
      <!-- color theme for nbeditor-settings-ColoringType -->
        <folder name="FontsColors">
            <folder name="Twilight">
                <folder name="Defaults">
                    <file name="org-netbeans-modules-defaults-highlight-colorings.xml" url="Twilight/editor.xml">
                        <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.defaults.Bundle"/>
                        <attr name="nbeditor-settings-ColoringType" stringvalue="highlight"/>
                    </file>
                    <file name="org-netbeans-modules-defaults-token-colorings.xml" url="Twilight/defaults.xml">
                        <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.defaults.Bundle"/>
                    </file>
                </folder>
            </folder>
            <folder name="EmacsStandard">
                <folder name="Defaults">
                    <file name="org-netbeans-modules-defaults-token-colorings.xml" url="EmacsStandard/defaults.xml">
                        <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.defaults.Bundle"/>
                    </file>
                </folder>
            </folder>
        </folder>

        <folder name="text">
            <folder name="x-erlang">
                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                <file name="language.instance">
                    <attr name="instanceCreate" methodvalue="org.netbeans.modules.erlang.editor.lexer.ErlangTokenId.language"/>
                    <attr name="instanceOf" stringvalue="org.netbeans.api.lexer.Language"/>
                </file>
                
                <!-- TODO - this should not be necessary; I'm doing this now to work around
                    bugs in color initialization -->
                <folder name="FontsColors">
                    <folder name="NetBeans">
                        <folder name="Defaults">
                            <file name="coloring.xml" url="fontsColors.xml">
                                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                            </file>
                        </folder>
                    </folder>
                    <folder name="Twilight">
                        <folder name="Defaults">
                            <file name="coloring.xml" url="Twilight/fontsColors.xml">
                                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                            </file>
                        </folder>
                    </folder>
                    <folder name="EmacsStandard">
                        <folder name="Defaults">
                            <file name="coloring.xml" url="EmacsStandard/fontsColors.xml">
                                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                            </file>
                        </folder>
                    </folder>
                </folder>

                <folder name="CodeTemplates">
                    <folder name="Defaults">
                        <file name="codeTemplates.xml" url="codeTemplates.xml">
                            <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                        </file>
                    </folder>
                </folder>
                <folder name="Keybindings">
                    <folder name="NetBeans">
                        <folder name="Defaults">
                            <file name="org-netbeans-modules-erlang-editor-keybindings.xml" url="keyBindings.xml"/>
                        </folder>
                    </folder>
                </folder>
            </folder>
        </folder>
    </folder>

    <folder name="CslPlugins">
        <folder name="text">
            <folder name="x-erlang">
                <file name="language.instance">
                    <attr name="instanceClass" stringvalue="org.netbeans.modules.erlang.editor.ErlangLanguage"/>
                </file>
            <!--file name="structure.instance">
               <attr name="instanceClass" stringvalue="org.netbeans.modules.scala.editing.ScalaStructureAnalyzer"/>
            </file-->
            </folder>
        </folder>
    </folder>
    
    <folder name="Loaders">
        <folder name="text">
            <folder name="x-erlang">
                <attr name="SystemFileSystem.icon" urlvalue="nbresloc:/org/netbeans/modules/erlang/editor/resources/Erlang.png"/>
                <attr name="iconBase" stringvalue="org/netbeans/modules/erlang/editor/resources/Erlang.png"/>
                <folder name="Actions">
                    <file name="OpenAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.OpenAction"/>
                        <attr name="position" intvalue="100"/>
                    </file>
                    <file name="Separator1.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="200"/>
                    </file>
                    <file name="CutAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.CutAction"/>
                        <attr name="position" intvalue="300"/>
                    </file>
                    <file name="CopyAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.CopyAction"/>
                        <attr name="position" intvalue="400"/>
                    </file>
                    <file name="PasteAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.PasteAction"/>
                        <attr name="position" intvalue="500"/>
                    </file>
                    <file name="Separator2.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="600"/>
                    </file>
                    <file name="NewAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.NewAction"/>
                        <attr name="position" intvalue="700"/>
                    </file>
                    <file name="DeleteAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.DeleteAction"/>
                        <attr name="position" intvalue="800"/>
                    </file>
                    <file name="RenameAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.RenameAction"/>
                        <attr name="position" intvalue="900"/>
                    </file>
                    <file name="Separator3.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="1000"/>
                    </file>
                    <file name="SaveAsTemplateAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.SaveAsTemplateAction"/>
                        <attr name="position" intvalue="1100"/>
                    </file>
                    <file name="Separator4.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="1200"/>
                    </file>
                    <file name="FileSystemAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.FileSystemAction"/>
                        <attr name="position" intvalue="1300"/>
                    </file>
                    <file name="Separator5.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="1400"/>
                    </file>
                    <file name="ToolsAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.ToolsAction"/>
                        <attr name="position" intvalue="1500"/>
                    </file>
                    <file name="PropertiesAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.PropertiesAction"/>
                        <attr name="position" intvalue="1600"/>
                    </file>
                </folder>
            </folder>
        </folder>
    </folder>
</filesystem>

Now build your new module, lunch it, and open a .erl file, you get:

Click on the picture to enlarge it

nn

Erlang Plugin for NetBeans in Scala#1: Scala Enumeration Implemented a Java Interface

ErlyBird (previous plugin) was written in Java, and based on NetBeans' Generic Languages Framework ( Project Schliemann). As NetBeans 7.0 introducing new  Parsing API, I'm going to migrate Erlang plugin for NetBeans to it, and rewrite in Scala.

Defining a Scala/Java mixed module project in NetBeans' trunk tree is a bit trick, but, since it's plain ant-based project, you can always take over it by writing/modifying build.xml. Challenges are actually from integrating Scala's Class/Object/Trait to an already existed framework. I met this kind of challenge quickly when I tried to implement an enum like class which should implement a Java interface org.netbeans.api.lexer.TokenId?

The signature of TokenId? is:

public interface TokenId {
    String name();
    int ordinal();
    String primaryCategory();
}

From the signature, you can get the hint to implement it as a Java enum. That's true and straightforward in Java. But how about in Scala?

There is an abstract class  Enumeration in Scala's library. The example code shows how to use it by extending it as a singleton Object. But for my case, I have to implement name() and ordinal() methods of TokenId? in the meantime, which brings some inconveniences for get the name() method automatically/magically for each enumeration value.

A quick google search gave some discussions about it, for example, a method reflection and invocation may be a choice to get name() simply implemented. I tried it, but finally dropped, because you cannot guarantee the condition of call stack of name(), it may happen to be called in this Object's own method, which then, may bring infinite cycled calls.

My final code is like:

package org.netbeans.modules.erlang.editor.lexer

object ErlangTokenId extends Enumeration {
  type ErlangTokenId = V

  // Extends Enumeration's inner class Val to get custom enumeration value
  class V(val name:String, val fixedText:String, val primaryCategory:String) extends Val(name) with TokenId {
    override
    def ordinal = id
  }
  object V {
    def apply(name:String, fixedText:String, primaryCategory:String) = new V(name, fixedText, primaryCategory)
  }
  
  val IGNORED = V("IGNORED", null, "ingore")
  val Error = V("Error", null, "error")

  // --- Spaces and comments
  val Ws = V("Ws", null, "whitespace")
  val Nl = V("Nl", null, "whitespace")
  val LineComment = V("LineComment", null, "comment")
  val CommentTag = V("CommentTag", null, "comment")
  val CommentData = V("CommentData", null, "comment")

  // --- Literals
  val IntegerLiteral = V("IntegerLiteral", null, "number")
  val FloatingPointLiteral = V("FloatingPointLiteral", null, "number")
  val CharacterLiteral = V("CharacterLiteral", null, "char")
  val StringLiteral = V("StringLiteral", null, "string")

  // --- Keywords
  val Andalso = V("Andalso", "andalso", "keyword")
  val After = V("After", "after", "keyword")
  val And = V("And", "and", "keyword")
  val Band = V("Band", "band", "keyword")
  val Begin = V("Begin", "begin", "keyword")
  val Bnot = V("Bnot", "bnot", "keyword")
  val Bor = V("Bor", "bor", "keyword")
  val Bsr = V("Bsr", "bsr", "keyword")
  val Bxor = V("Bxor", "bxor", "keyword")
  val Case = V("Case", "case", "keyword")
  val Catch = V("Catch", "catch", "keyword")
  val Cond = V("Cond", "cond", "keyword")
  val Div = V("Div", "div", "keyword")
  val End = V("End", "end", "keyword")
  val Fun = V("Fun", "fun", "keyword")
  val If = V("If", "if", "keyword")
  val Not = V("Not", "not", "keyword")
  val Of = V("Of", "of", "keyword")
  val Orelse = V("Orelse", "orelse", "keyword")
  val Or = V("Or", "or", "keyword")
  val Query = V("Query", "query", "keyword")
  val Receive = V("Receive", "receive", "keyword")
  val Rem = V("Rem", "rem", "keyword")
  val Try = V("Try", "try", "keyword")
  val When = V("When", "when", "keyword")
  val Xor = V("Xor", "xor", "keyword")

  // --- Identifiers
  val Atom = V("Atom", null, "identifier")
  val Var = V("Var", null, "identifier")

  // --- Symbols
  val LParen = V("LParen", "(", "separator")
  val RParen = V("RParan", ")", "separator")
  val LBrace = V("LBrace", "{", "separator")
  val RBrace = V("RBrace", "}", "separator")
  val LBracket = V("LBracket", "[", "separator")
  val RBracket = V("RBracket", "]", "separator")
  val Comma = V("Comma", ",", "separator")
  val Dot = V("Dot", ".", "separator")
  val Semicolon = V("Semicolon", ";", "separator")
  val DBar = V("DBar", "||", "separator")
  val Bar = V("Bar", "|",  "separator")
  val Question = V("Question", "?","separator")
  val DLt = V("DLt", "<<", "separator")
  val LArrow = V("LArrow", "<-", "separator")
  val Lt = V("Lt", "<", "separator")
  val DGt = V("DGt",  >", "separator")
  val Ge = V("Ge",  =", "separator")
  val Gt = V("Gt",  ", "separator")
  val ColonMinus = V("ColonMinus", ":-", "separator")
  val DColon = V("DColon", "::", "separator")
  val Colon = V("Colon", ":", "separator")
  val Hash = V("Hash", "#", "separator")
  val DPlus = V("DPlus", "++", "separator")
  val Plus = V("Plus", "+", "separator")
  val DMinus = V("DMinus", "--", "separator")
  val RArrow = V("RArrow", "->", "separator")
  val Minus = V("Minus", "-", "separator")
  val Star = V("Star", "*", "separator")
  val Ne = V("Ne", "/=", "separator")
  val Slash = V("Slash", "/", "separator")
  val EEq = V("EEq", "=:=", "separator")
  val ENe = V("ENe", "=/=", "separator")
  val DEq = V("DEq", "==", "separator")
  val Le = V("le", "=<", "separator")
  val Eq = V("Eq", "=", "separator")
  val Exclamation = V("Exclamation", "!", "separator")
}

First, I defined a class V which extends Enumeration.Val, and implemented TokenId with an extra field: fixedText.

Then, I have to explicitly put the value's name to this class and pass it to Enumeration.Val's constructor, so function ErlangTokenId.valueOf(String) will work as Java's enum type.

By type ErlangTokenId = V, type ErlangTokenId.V is now aliased as ErlangTokenId, so you can use ErlangTokenId instead of ErlangTokenId.V everywhere now, which exactly gets the effect of one of the behaviors of Java's enum: enum's value is the same type of enum itself.

Scala Plugin 0.15.1 for NetBeans Released

I'm pleased to announce the availability of Scala plugin 0.15.1 for NetBeans 6.5.

  • Bundling with Scala 2.7.3
  • Partly supported Java/Scala mixed project: building project is OK (need to create a new Scala project); Java source is visible in Scala editor.
  • Various bugs fixes

To download, please go to: https://sourceforge.net/project/showfiles.php?group_id=192439&package_id=256544&release_id=654650

For more information, please see http://wiki.netbeans.org/Scala

Bug reports are welcome.

=== Important Notice ===:

If you have a previous Scala plugin installed, after new plugins installed, you need to locate the NetBeans user profile:

  • On Windows system, the default location is: "C:\Documents and Settings\<username>\.netbeans\<version-number>\"
  • On Unix/Linux/MaxOS system: "<username>/.netbeans/<version-number>/"
There is directory "scala" under above location, you should delete the whole sub-folder "scala-2.7.2.final", and leave only "scala-2.7.3.final".

=====================

NetBeans on OpenSolaris 08.11 in VirtualBox in Mac OS

My Macbook is ân old one with Mac OS X 10.4, I have no way to get Java 6. I´ve tracked OpenSolaris? for a long time, with OpenSolaris? 08.11 is going to be released, I think I should have a try to see if I can do my daily work on OpenSolaris? rather than Mac OS on my MacBook?.

I then installed VirtualBox? on my MacOS, then a guest OpenSolaris? with 1024M memory and 16G disk. I downloaded and installed Java JDK 6 and NetBeans 6.5, plus my Scala plugins.

It rocks seamlessly. The only problem is, should I re-format my harddisk and install OpenSolaris? instead of Mac OS now?

There are a lot of guesses on Sun´s future these days, but, with all those innovations from Sun or taken by Sun, Why cloud-computing? Should still be sunshine-computing.

Click on the picture to enlarge it

MyScreen_081122.png

Scala Plugin for Coming NetBeans 6.5 Official Release

I'm pleased to announce the availability of Scala plugin for coming NetBeans 6.5 official release.

  • Much better code-completion
  • Two new color themes: Twilight and Emacs Standard
  • Various bugs fixes
  • It's not perfect, but fairly stable
  • Works good with NetBeans Maven plugin

To download, please go to: https://sourceforge.net/project/showfiles.php?group_id=192439&package_id=256544&release_id=641359

For more information, please see http://wiki.netbeans.org/Scala

Bug reports are welcome.

It works also on NetBeans RC2. If you have previous version of Scala plugin installed, you can upgrade to this version.

Scala for NetBeans Screenshot#15: Twilight Color Theme

>>> Updated Nov 11:

Emacs Standard color theme already be there.

======

I'm beginning to write some real thing based on Liftweb. The more code I wrote, the more bugs of Scala plugin were fixed.

Not only bugs are being fixed, I also created a Twilight color theme for this plugin. And an Emacs color theme is also on the road.

When NetBeans 6.5 is official released, I'll put a new Scala plugin too.

Click on the picture to enlarge it

ScalaEditor_081109.png

ScalaEditor_081110.png

New Scala Plugin for NetBeans 6.5 RC2 Is Packed

I packed a new Scala plugin, and tried several times to upload it to NetBeans' Plugins Portal and could not get job successfully done. So I uploaded it to sourceforge.net instead. The zip file is located at: [ https://sourceforge.net/project/showfiles.php?group_id=192439&package_id=256544&release_id=638797 scala-plugin-081106.zip

NetBeans' trunk has been targeting 7.0 now, the plugin on Last Development Build update center is no longer compatible with 6.5 RC2. That is, you should ignore my previous blog talked about installing plugin on 6.5 RC2, if you are using 6.5 RC2, you should download and install this one.

Nightly built version users can get latest plugin via NetBeans' plugin management feature as normal.

For more information, please visit  http://wiki.netbeans.org/Scala

This version fixed various bugs, and with some enhancements. The bundling Scala runtime is 2.7.2RC6, with NetBeans' maven plugin, it works on newest  Liftweb project.

Bug reports are always welcome.

Click on the picture to enlarge it

ScalaEditor_081106.png

Install Scala Plugin for NetBeans 6.5 RC2

>>> Updated Nov 8
Content of this blog is out of date, please look at New Scala Plugin for NetBeans 6.5 RC2 Is Packed
===

NetBeans 6.5 RC2 released. The Scala plugin on NetBeans' Plugins Portal is not compilable with RC2. To get Scala plugin working with RC2, do:

# Open NetBeans 6.5 RC2, go to "Tools" -> "Plugins", check "Setting" -> "Add", add new update center as "Last Development Build" with url: http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastStableBuild/artifact/nbbuild/nbms/updates.xml.gz

# Then in the "Available Plugins" tab, you can find the "Scala" category (or, you can click on "Name" in "Available Plugins" tab to find them. You may need to click "Reload Catalog" to get the latest available modules), check "Scala Kit" and click "Install", following the instructions. Restart IDE.

I'll re-pack a new version of Scala plugin for Plugins Portal when NetBeans 6.5 is officially released.

Scala for NetBeans Screenshot#14: Refined Color Theme

With more Scala coding experience, I refined color theme of Scala plugin for NetBeans. By desalinating Type name a bit, I found I can concentrate on logic a bit better. And all function calls are highlighted now, so, for multiple-line expression, when error-prone op is put on wrong new line, you can get some hints at once. It also gives you hint if a val/var is with implicit calling, which will be highlighted as a function call.

There are still some bugs when infer var/val's correct type in some cases.

Now the editor is much informative with highlighting and bubble pop display of type information (move mouse on it with CTRL/COMMAND pressed).

You'll need to update to newest 1.9.0 Editing module, please wait for it appealing on Development Update Center.

Click on the picture to enlarge it

nn

Scala for Netbeans Beta Is Ready, Working with NetBeans 6.5 Beta

>>> Updated Aug 15:

For Windows Vista users: There is a known bug  #135547 that may have been fixed in trunk but not for NetBeans 6.5 Beta, which causes exception of "NullPointerException at org.openide.filesystems.FileUtil.normalizeFileOnWindows" when create Scala project. If you are Vista user and like to have a try on Scala plugins, you may need to download a recent nightly build version of NetBeans. Since I have none Vista environment, I'm not sure about above message.

======

I'm pleased to announce that the first beta of Scala for NetBeans is released, followed NetBeans 6.5 beta releasing. The availability and installation instructions can be found at  http://wiki.netbeans.org/Scala.

Features:

  • Full featured Scala editor
    • syntax and semantic coloring
    • outline navigator
    • code folding
    • mark occurrences
    • go to declaration
    • instant rename
    • indentation
    • formatting
    • pair matching
    • error annotations
    • code completion
  • Project management (build/run/debug project)
  • Debugger
  • Interactive console
  • JUnit integration
  • Maven integration (works with [ http://www.liftweb.net Lift Web Framework)

There are some  known issues. Bug reports are welcome.

Installation on NetBeans 6.5 beta:

button, choose the directory where the Scala plugins are unzipped, select all listed *.nbm files, following the instructions. Restart IDE.

Run/Debug Lift Web App Using Scala/Maven Plugin for NetBeans

I'm a newbie to Maven, so I encountered some issues when run/debug Lift apps. The following are tips that I got, it may not be perfect, but at least work.

1. When pom.xml is changed, and classes not found errors happen on editor, you can close and reopen the project

This is a known issue, that I didn't refresh compiling context when pom.xml changed, thus the classpath dependency may be not refreshed at once, I'll fix this issue in the near future.

2. Run project by external Maven instead of embedded Maven of NetBeans plugin

I encountered java.lang.NoClassDefFoundError org/codehaus/plexus/util/DirectoryScanner when use embedded NetBeans Maven plugin (3.1.3) when invoke "Run" project, but fortunately, you can custom the binding actions in NetBeans Maven. The steps are:

  • Right click project node, choose "Properties"
  • Click on "Actions" in left-pane, choose "Use external Maven for build execution", and "set external Maven home"
  • Choose "Run project" in right-pane, input "jetty:run"
  • Choose "Debug project" in right-pane, input "jetty:run"

3. How to debug project

I'm a bit stupid here, since I don't like to change MAVEN_OPTS frequently. So I choose to do:

$ cd $MAVEN_HOME$/bin
$ cp mvn mvn.run
$ cp mvnDebug mvn

Then I invoke "Debug" action from NetBeans toolbar, and get NetBeans' output window saying:

WARNING: You are running Maven builds externally, some UI functionality will not be available.
Executing:/Users/dcaoyuan/apps/apache-maven-2.0.9/bin/mvn jetty:run
Preparing to Execute Maven in Debug Mode
Listening for transport dt_socket at address: 8000

Open menu "Debug" -> "Attach Debugger...", in the popped window, for "Port:", input "8000". Everything goes smoothly then. You add/remove breakpoints just as you are doing for a regular Scala project.

Of course, if you want to turn back to "Run" from "Debug", you have to "cp mvn.run mvn" back.

Anybody can give me hints on how to get this setting simple? in NetBeans Maven plugin.

Here's a snapshot: (click to enlarge it)

ScalaEditor_080729a.png

Scala, NetBeans, Maven, and yes, Lift now

Per recent changes, Scala for NetBeans can live with Maven for NetBeans, and yes,  Lift web framework.

The Maven for NetBeans has done an excellent work for Maven's project integration, with proper classpath supporting and indexing. So the Scala editor is well aware of auto-completion, types etc.

Here's a snapshot: (click to enlarge it)

ScalaEditor_080729.png

To get above features, you have to download the latest NetBeans daily build, then install Scala's plugins and Maven plugins.

For Scala plugins, see  http://wiki.netbeans.org/Scala; For Maven plugins, see  http://wiki.netbeans.org/MavenBestPractices.

Implementation of Scala for NetBeans based on GSF

The Scala for NetBeans is under pre-beta stage, other than bug-fixes, I'm preparing some documentations for it too. If you are interested in how to write language supporting under GSF's framework, you can take a look at this working documentation.

Implementation of Scala for NetBeans

And also:

Proposal of Scala for NetBeans

Progressing of Scala for NetBeans

Scala for NetBeans Screenshot#12: JUnit integration

There is a Scala JUnit Test template in Scala for NetBeans now, you can create Scala JUnit testing and run testing, see testing result.

Here is an example test file:

 /*
 * DogTest.scala
 *
 * To change this template, choose Tools | Template Manager
 * and open the template in the editor.
 */

package scalajunit

import org.junit.After
import org.junit.Before
import org.junit.Test
import org.junit.Assert._

class DogTest {

    @Before
    def setUp() = {
    }

    @After
    def tearDown() = {
    }

    @Test
    def testHello() = {
        val dog = new Dog()
        dog.talk("Mr. Foo")
        assert(true)
    }

    @Test
    def testAdding() = {
        assertEquals(4, 2 + 3)
    }
}

To get JUnit working, upgrade to Scala Project Module version 1.2.14, and re-create a new project for exist old project (if there has been one). Under the "Test Packages", create your testing packages, right click on package node, select create "Scala JUnit Test".

To run tests, right-click on project node, choose "Test".

And also, the beta-release is feature frozen, I'll concentrate on bug-fixes in next 1-2 weeks, and get it ready for public release as beta.

nn

Employ Scala's Native Compiler in Scala for NetBeans

>>> Updated 3 hours later
Fixed document displaying and go to declaration for Java element.
Please update to Scala Editing version 1.1.4
======

Well, it's a fairly long time after I latest blog about Scala for NetBeans. I was busy on several things, and can only work on this project on my spare time.

The good news is that I've integrated Scala's native compiler into this handy plugin, it means that the error messages shown in the editor will be the same as building now. And, the auto-completion feature is totally rewritten too, which also use AST tree that was created by Scala's native compiler to get all the candidate content assistant items.

The only problem is that the Java's document comments and offset to be go to when press CTRL + Click does not work properly yet, I have several way to resolve it, but I'm looking for a best way.

Another exciting news is that I've been invited to join NetBeans Dream Team, and of course, I accepted this invitation.

I'm planning to get Scala plugin for NetBeans to beta release in August, which will be compatible with NetBeans 6.5

Parsing Performance of Scala for NetBeans

I'm re-considering the indexing mechanism of Scala for NetBeans. The indexing is used to store meta-info of parsed templates/methods etc, for auto-completion and document/source offsets searching. Currently, the parsing phases include lexer, syntax and semantic analysis, not include type inference and type check.

With a basic performance testing on all Scala standard library and liftweb's library, the maxima parsing time seems less than 1s, the average parsing time is around 0.2s, not bad.

So, this may let the indexing feature a lot simple, I can store classes/objects/traits' meta-info only, instead of including their type parameters and their members (fields/methods, scoped importing etc), these additional information can be got via re-parsing the source file or querying the class file.

Bundled Latest Scala Runtime to Scala for NetBeans

I just bundled latest Scala runtime to Scala for NetBeans, the version is 2.7.1.final, this brings two things:

First, you do not need to set SCALA_HOME any more to get whole plugins working. But, you can still set SCALA_HOME to specifying the target Scala version, if so, you should also need to download and unzip source jars to $SCALA_HOME/src;

Second, I'll begin to write some code in Scala instead of Java for Scala plugins. I can evaluate the features of Scala plugins in daily work, find and fix more bugs of plugins.

Scala for NetBeans Screenshot#12: Better Completion with More Types Inferred

>>> Updated May 11
Mostly, infix expressions can be type inferred now. CTRL+Click on infix op name, it will bring you to declaration (since 1.0.29.0)
===

Well, the type inference work is not so easy (with performance in mind), but anyway, I've got a bit more progress, at least, the chained member call can now be correctly inferred in a lot of cases. It's some level as Tor's JavaScript for NetBeans now.

First, let's create a val "node", which is a "scala.xml.Node"

nn

Then, input '.' to invoke completion, as I know which type is of "node", the proposal items look good.

nn

I choose "descendant" function (which returns a "List"), and input '.' again, we can see the proposal items look still good.

nn

These features also work on Java's class.

Known issues:

  • It seems the indexing/scanning for Scala standard library source will perform twice when you first installed Scala plugins
  • The type inference is not consistence yet, so don't be strange for the strange behavior sometimes

Again, don't forget to download scala standard library's source jars and unzip to $SCALA_HOME/src, per sub-folder per jar

Scala for NetBeans Screenshot#11: Go to Remote Declaration and Doc Tooltip

Two crazy days of my spare time, I was coding between sleeping and eating, with a lot of cups of coffee. Now Scala plugins support Go-To remote declarations (CTRL + Click on var/function name), and when you put cursor on the identifier name with CTRL pressed, or under auto-completion, the doc will also be shown as tooltip.

These features work for Java classes too with a bit poor performance, I'll fix it later (fixed).

Not all identifiers have been type inferred, so these features are not applicable for all identifiers.

Please update to Scala Editing module as version 1.0.26.xxx when it's available, which is the only stable one these days. Remember to unzip Scala lib's source under $SCALA_HOME/src

nn

nn

Scala for NetBeans Screenshot#10: Working on Auto-Completion for Java

I've done some hacking work to get Java classes to be completion enabled, but it's not full functional. Any way, it's a good start point for auto-completion for Scala plugins, I hope to get more type inference work to be finished, and finally support both Scala/Java classes smart-completion.

Click on the picture to enlarge it

nn

nn

Scala for NetBeans Screenshot#9: Working on Auto-Completion

With the indexed cache of project's Class/Trait/Object, and Scala standard library's source files, the auto-completion is a bit smarter now. If the val/var is defined with type, the auto-completion can know which methods will be suggested. (Not work for java classes yet)

To get this working, you should follow these steps:

  • Update to newest Scala plugins (Editing version 1.0.21.1)
  • Delete the old-cache files which are located at your NetBeans's configuration directory (for example, .netbeans/dev/var/cache).
  • Download Scala standard library's source file, unzip them to $SCALA_HOME/src, per sub-folder per source jar file

Click on the picture to enlarge it

nn

New Scala Plugins for NetBeans are Available for Public Test, and Fortress, Erlang

>>> Updated Apr 22
Due to the incompatibly changes of NetBeans underlaying modules, Scala plugins can not be installed/updated on NetBeans 6.1 any more, you should get the latest nightly build to play with Scala plugins.
===

>>> Updated Apr 19

  • Fixed some broken syntax
  • Fixed NPE caused by brace completion
  • Fixed indentation of case class/object
  • Added formatting options ("Preference" -> "Scala" -> "Formatting")
Please update your Scala Editing plugin to version 1.0.19 which will be available after NetBeans hudson building.
===

The new written Scala plugins for NetBeans are available for public test now, which can be installed on NetBeans 6.1 RC, and latest NetBeans nightly build. To get start, please visit http://wiki.netbeans.org/Scala

The following features are ready for test:

  • Syntax highlighting
  • Auto-indentation
  • Brace completion
  • Formatter
  • Outline navigator
  • Occurrences mark for local variables and functions
  • Instance rename for local variables and functions
  • Go-to-declaration for local variables and functions
  • Scala project
  • Basic debugger

And with known issues:

  • Auto-completion it not fully supported yet and not smart
  • There is no parsing errors recovering yet
  • Semantic errors are not checked on editing, but will be noticed when you build project
  • Due to the un-consistent of Scala's grammar reference document, there may be some syntax broken issues

BTW, Fortress editing plugin is also available on "Last Development Build" update center, see the installation part of http://wiki.netbeans.org/Scala to get it installed. It's a very alpha stage plugin.

And, Erlang plugins are also available from "Last Development Build" update center too, that is, you can install and use Erlang plugins with Ruby, Scala, JavaScript on the same NetBeans IDE (6.1 RC or nightly build). Thanks to Tor's work, the indexing performance has been improved a lot.

Erlang plugin will be rewritten in the near future too.

Scala for NetBeans Screenshot#8: Working on Indexer

I've done some basic indexer code, that is, all source files under a project will be parsed, analyzed, then indexed (class/object/trait, functions, fields etc). But it's just a start, before I finished type inference, if you press CTRL+SPACE to invoke completion, there are a lot of indexed Class/Object/Trait/Function will be roughly shown on you :-), it's not smart, it's more like a puzzle, you should decide which one is applicable by yourself. But you can get a view of the coming completion feature.

Click on the picture to enlarge it

nn

Progress of Scala for NetBeans - with a New Written Lexer and Parser

According to this post:

I talked to Jetbrains about this, and they told me that they stopped working on the Scala plugin for the time being, because - demand for Groovy/Ruby was higher - the language was moving too fast - Scala is a terribly difficult language for compiler/tool writers, and the only good way to analyze Scala programs might be through the official compiler, which didn't yet support this

It's true that "Scala is a terribly difficult language for compiler/tool writers", but I'm trying to bypass "the only good way to analyze Scala programs might be through the official compiler"

Before rewriting Scala for NetBeans, I considered some parser choices, one was Scala's native compiler, which is good for compiling/building Scala project, but not suitable for Editor. And JavaCC, ANTLR, which may be good enough, but it's not natural to express Scala's grammar.

Then I found Rats! which is used by Fortress, a very very clean, powerful parser generator. After couple of days working, I got an incremental lexer for Scala, and a parser for Scala that with Scala's grammar being naturally expressed (the grammar definition is ParserScala.rats). The benefit of a complete controllable parser is that I can now do some type inference and wholly semantic analysis freely and immediately.

Another progress is that I've decoupled the Scala project's dependency on Java.source's classpath in NetBeans, instead, GSF's classpath is used in Scala project module now. That means, I can begin the indexer for Scala's standard library and project source files.

The next steps will be type inference; smart completion with type inferred information; indexer for later refectory and usages searching; parsing error recover etc.

nn

Where We Are - Rewriting Scala for NetBeans

>>> Updated:
The rewritten plugin will be available, depending on the NetBeans' nightly build, it may be available after 10 hours or more. New features include a formatter (CTRL+SHIFT+F), and better brace completer. Smart auto-completion does not work now, needs further working.
===

The new Scala for NetBeans is going to form a good shape, I've got syntax highlighting, indentation, formatting, brace matching, and basic structure outline working. Here's a snapshot showing complex Scala statements being highlighted and reformatted properly:

nn

Begin Rewriting Scala for NetBeans

I'm re-writing Scala for NetBeans. Everything is broken except syntax highlighting.

Please be patient to wait for further progress.

Developing IDE Based on GSF for NetBeans#1 - Minimal Support

There has been GSF (Generic Scripting Framework) which is Tor's working derived and abstracted from Java supporting code, and, the base of Ruby/JavaScript support for NetBeans.

So, how to develop an IDE based on GSF for NetBeans? I'd like to share some experiences in this series of articles, a series of outline description, without too much code and details, for detailed information, please go into the source code on hg.netbeans.org

I. Minimal Support - Highlighting

To implement a minimal support of your editor, you need to implement/extend following classes/interface:

public class ScalaLanguage implements GsfLanguage
public class ScalaMimeResolver extends MIMEResolver
public enum ScalaTokenId implements TokenId
public class ScalaLexer implements Lexer<ScalaTokenId>

Where ScalaLexer is the token scanner of your language.

Then, register your language in layer.xml:

<filesystem>
    <folder name="Editors">
        <folder name="text">
            <folder name="x-scala">
                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.scala.editing.Bundle"/>

                <file name="language.instance">
                    <attr name="instanceCreate" methodvalue="org.netbeans.modules.scala.editing.lexer.ScalaTokenId.language"/>
                    <attr name="instanceOf" stringvalue="org.netbeans.api.lexer.Language"/>
                </file>
                
                <folder name="FontsColors">
                    <folder name="NetBeans">
                        <folder name="Defaults">
                            <file name="coloring.xml" url="fontsColors.xml">
                                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.scala.editing.Bundle"/>
                            </file>
                        </folder>
                    </folder>
                </folder>
                
                <folder name="CodeTemplates">
                    <folder name="Defaults">
                        <file name="codetemplates.xml" url="codetemplates.xml">
                            <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.scala.editing.Bundle"/>
                        </file>
                    </folder>
                </folder>

                <folder name="Keybindings">
                    <folder name="NetBeans">
                        <folder name="Defaults">
                            <file name="org-netbeans-modules-scala-editing-keybindings.xml" url="DefaultKeyBindings.xml"/>
                        </folder>
                    </folder>
                </folder>
            </folder>
        </folder>
    </folder>
      
    <folder name="GsfPlugins">
        <folder name="text">
            <folder name="x-scala">
                <file name="language.instance">
                    <attr name="instanceOf" stringvalue="org.netbeans.modules.gsf.api.GsfLanguage"/>
                    <attr name="instanceClass" stringvalue="org.netbeans.modules.scala.editing.ScalaLanguage"/>
                </file>
            </folder>
        </folder>
    </folder>
    
    <folder name="Loaders">
        <folder name="text">
            <folder name="x-scala">
                <attr name="SystemFileSystem.icon" urlvalue="nbresloc:/org/netbeans/modules/scala/editing/resources/scala16x16.png"/>
                <attr name="iconBase" stringvalue="org/netbeans/modules/scala/editing/resources/scala16x16.png"/>
                <folder name="Actions">
                    <file name="OpenAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.OpenAction"/>
                        <attr name="position" intvalue="100"/>
                    </file>
                    <file name="Separator1.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="200"/>
                    </file>
                    <file name="CutAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.CutAction"/>
                        <attr name="position" intvalue="300"/>
                    </file>     
                    <file name="CopyAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.CopyAction"/>
                        <attr name="position" intvalue="400"/>
                    </file>
                    <file name="PasteAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.PasteAction"/>
                        <attr name="position" intvalue="500"/>
                    </file>
                    <file name="Separator2.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="600"/>
                    </file>
                    <file name="NewAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.NewAction"/>
                        <attr name="position" intvalue="700"/>
                    </file>
                    <file name="DeleteAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.DeleteAction"/>
                        <attr name="position" intvalue="800"/>
                    </file>
                    <file name="RenameAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.RenameAction"/>
                        <attr name="position" intvalue="900"/>
                    </file>
                    <file name="Separator3.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="1000"/>
                    </file>
                    <file name="SaveAsTemplateAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.SaveAsTemplateAction"/>
                        <attr name="position" intvalue="1100"/>
                    </file>
                    <file name="Separator4.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="1200"/>
                    </file>
                    <file name="FileSystemAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.FileSystemAction"/>
                        <attr name="position" intvalue="1300"/>
                    </file>
                    <file name="Separator5.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="1400"/>
                    </file> 
                    <file name="ToolsAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.ToolsAction"/>
                        <attr name="position" intvalue="1500"/>
                    </file> 
                    <file name="PropertiesAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.PropertiesAction"/>
                        <attr name="position" intvalue="1600"/>
                    </file> 
                </folder>            
            </folder>
        </folder>
    </folder>
</filesystem>

Don't forget to prepare all these resource files that registered in above layer.xml, such as scala16x16.png etc

After that, write an one-line service descriptor org.openide.filesystems.MIMEResolver under META-INF/services, which looks like

org.netbeans.modules.scala.editing.ScalaMimeResolver

That's it.

Fortress for NetBeans Screenshot#2: Mark Occurrences and Instant Rename

Well, I'm going on Fortress for NetBeans, and quickly get some basic Mark-Occurrences working, and of course, Instant-Rename .

As more thoughts are proved, I may begin new Scala editor module soon.

nn

Fortress for NetBeans Screenshot#1: Syntax Highlighting

>>> Updated Mar 19:
The outline navigator will do some unicode rendering; Brace matching works.
===

I'm planning to re-write Scala editor module, based on a new parser (maybe Tor from Sun and Carl from Google will help this project too). But before that, I'd like to prove some thoughts. That comes Fortress for NetBeans.

To know more about Fortress, please go to http://projectfortress.sun.com/Projects/Community/

Fortress's grammar is expressed in PEG (Parsing Expression Grammars), and is implemented on Rats!, a clean PEG parser generator.

I got some interesting experiences on how to let Rats! become a simple lexical token stream generator, of course, it has some limits, but these lexical tokens are used only for basic/quick highlighting, brace-matching etc, I do not use it for syntax parsing and semantic analysis, so the limits are not an effect. The syntax parsing, semantic analysis and grammar checking will be from original Fortress parser.

Here's a screenshot of my experimental work:

nn

Actually I've done a bit of semantic works too, thus you can see the function definitions were bold highlighted and the outline navigator window was almost there.

ErlyBird 0.16.0 Released - An Erlang IDE based on NetBeans

I'm pleased to announce ErlyBird 0.16.0, an Erlang IDE based on NetBeans. This is an important feature release in size of 25M. If you have latest NetBeans nightly build installed, you can also install ErlyBird modules via update center.

CHANGELOG:

  • Project metadata file is changed, please see Notes
  • Instant rename (put caret on variable or function name, press CTRL+R, when rename finished, press ENTER)
  • Go-To-Declaration to macros that are defined included header files
  • Fixed: Go-To-Declaration to -inlcudelib won't work again after this include header file was opened in editor once
  • Fixed: syntax broken for packaged import attribute
  • Fixed: syntax broken for wild attribute
  • Completion suggestion will not search other projects
  • Track GSF changes, reindex performance was improved a lot; Can live with other GSF based language support now (Ruby, Groovy etc)

Java JRE 5.0+ is required.

To download, please go to: http://sourceforge.net/project/showfiles.php?group_id=192439

To install:

  1. Unzip erlybird-bin-0.16.0-ide.zip to somewhere.
  2. Make sure 'erl.exe' or 'erl' is under your environment path
  3. For Windows user, execute 'bin/erlybird.exe'. For *nix user, 'bin/erlybird'.
  4. Check/set your OTP path. From [Tools]->[Options], click on 'Erlang', then 'Erlang Installation' tab, fill in the full path of your 'erl.exe' or 'erl' file. For instance: "C:/erl/bin/erl.exe"
  5. The default -Xmx option for jvm is set to 256M, ErlyBird now works good with less memory, such as -Xmx128M. If you want to increase/decrease it, please open the config file that is located at etc/erlybird.conf, set -J-Xmx of 'default_options'.

When run ErlyBird first time, the OTP libs will be indexed. The indexing time varies from 10 to 30 minutes deponding on your computer.

Notes:

  1. Since project metadata format is changed, to open old ErlyBird created project, you should modify project.xml which is located at your project folder: nbproject/project.xml, change line:
    <type>org.netbeans.modules.languages.erlang.project</type>
    
    to:
    <type>org.netbeans.modules.erlang.project</type>
    
  2. If you have previous version ErlyBird installed, you should delete the old cache files which are located at:
    • *nix: "${HOME}/.erlybird/dev"
    • mac os x: "${HOME}/Library/Application Support/erlybird/dev"
    • windows: "C:\Documents and Settings\yourusername\.erlybird\dev" or some where

The status of ErlyBird is still Alpha, feedbacks and bug reports are welcome.

Scala for NetBeans Recent Updates

I added a changelog page on http://wiki.netbeans.org/ScalaChangeLog, where you can learn the latest progressing.

Here is a summary of recent updates:

20080214 (Needs latest nightly build)

  • [Editing] fixed: highlighting of occurences is not properly refreshed

20080213 (Needs latest nightly build)

  • [Editing] fixed some issues of indent and brace-completion

20080212 (Needs latest nightly build)

  • [Editing] fixed: case pattern cause a parser error at the minus symbol before number. (Thanks to misterm for reporting)
  • [Project] sync with java.commonapi. Needs latest nightly build

20080205

  • [Editing] fixed some broken grammars. (Thanks to Michael Nischt for collecting)
  • [Project] fixed environment property, now scala.home property may not need to be set. (Thanks to Denis)

20080204

  • [Debugger] fixed exceptions when put cursor on code
  • [Debugger] "Add watches" should work now

Working with NetBeans Mercurial Repository

Just back from snowboarding. I tried to hg pull -uv to update to the latest code base of NetBeans, but got:

** unknown exception encountered, details follow
** report bug details to http://www.selenic.com/mercurial/bts
** or mercurial@selenic.com
** Mercurial Distributed SCM (version 0.9.4)
Traceback (most recent call last):
  File "/opt/local/bin/hg", line 11, in ?
    mercurial.commands.run()
  File "/opt/local/lib/python2.4/site-packages/mercurial/commands.py", line 3110, in run
    sys.exit(dispatch(sys.argv[1:], argv0=sys.argv[0]))
  File "/opt/local/lib/python2.4/site-packages/mercurial/commands.py", line 3107, in dispatch
    return cmdutil.runcatch(u, args, argv0=argv0)
  File "/opt/local/lib/python2.4/site-packages/mercurial/cmdutil.py", line 37, in runcatch
    return dispatch(ui, args, argv0=argv0)
  File "/opt/local/lib/python2.4/site-packages/mercurial/cmdutil.py", line 364, in dispatch
    ret = runcommand(ui, options, cmd, d)
  File "/opt/local/lib/python2.4/site-packages/mercurial/cmdutil.py", line 417, in runcommand
    return checkargs()
  File "/opt/local/lib/python2.4/site-packages/mercurial/cmdutil.py", line 373, in checkargs
    return cmdfunc()
  File "/opt/local/lib/python2.4/site-packages/mercurial/cmdutil.py", line 356, in 
    d = lambda: func(ui, repo, *args, **cmdoptions)
  File "/opt/local/lib/python2.4/site-packages/mercurial/commands.py", line 2063, in pull
    return postincoming(ui, repo, modheads, opts['update'])
  File "/opt/local/lib/python2.4/site-packages/mercurial/commands.py", line 2001, in postincoming
    return hg.update(repo, repo.changelog.tip()) # update
  File "/opt/local/lib/python2.4/site-packages/mercurial/hg.py", line 248, in update
    stats = _merge.update(repo, node, False, False, None, None)
  File "/opt/local/lib/python2.4/site-packages/mercurial/merge.py", line 541, in update
    checkunknown(wc, p2)
  File "/opt/local/lib/python2.4/site-packages/mercurial/merge.py", line 65, in checkunknown
    for f in wctx.unknown():
  File "/opt/local/lib/python2.4/site-packages/mercurial/context.py", line 413, in unknown
    def unknown(self): return self._status[4]
  File "/opt/local/lib/python2.4/site-packages/mercurial/context.py", line 369, in __getattr__
    self._status = self._repo.status()
  File "/opt/local/lib/python2.4/site-packages/mercurial/localrepo.py", line 864, in status
    list_ignored, list_clean)
  File "/opt/local/lib/python2.4/site-packages/mercurial/dirstate.py", line 445, in status
    for src, fn, st in self.statwalk(files, match, ignored=list_ignored):
  File "/opt/local/lib/python2.4/site-packages/mercurial/dirstate.py", line 421, in statwalk
    sorted_ = [ x for x in findfiles(f) ]
  File "/opt/local/lib/python2.4/site-packages/mercurial/dirstate.py", line 382, in findfiles
    if not ignore(np):
RuntimeError: internal error in regular expression engine

It was the second time I encountered this issue, I don't know why, so, I just upgraded my Mercurial from 0.9.4 to 0.9.5, and tried to get a new clone. I typed:

hg clone http://hg.netbeans.org

Everything seemed going smoothly, I had a new repository now. I copied my old hgrc to replace the un-configured one, and typed ant build, guess what? I got a lot of package javax.help does not exist, build failed. I knew that was because of that the binary files were not downloaded properly. It seemed I should leave the original hdrc there, and let ant hook the external.py. So I removed all extra lines except the original content of hgrc:

[paths]
default = http://hg.netbeans.org/main/

And tried ant build again, this time it went smoothly again.

LESSON: Do not modify hgrc before you do first "ant build", which will hook external.py and add "decode", "encode" automatically.

Scala for NetBeans: Debugger Modules are Available for Test

=== Updated Feb 5 ==>
I just fixed a defect of debugger modules, the updated modules will be available after new nightly build successfully. Since the underlaying changes of NetBeans APIs, you may need to download a new nightly build.
===

Scala Debugger modules are available for preview and test, which can be installed via Update Center for newest nightly build version of NetBeans (I tested on NetBeans IDE 6.1 200802040003). Debug feature includes two modules: Scala Debugger and Scala Debugger Projects Integration (you can click on "Name" in "Available Plugins" tab to find them), for more getting started information, please see http://wiki.netbeans.org/Scala

Google Group:

Known Issues:

  • "Run to cursor" does not work yet
  • "Step into Scala standard library's code" is not supported yet
  • "Add watches" is not supported yet
  • Complex condictions are not tested yet

Adventure with Nightly Build:

  • Nightly build version of NetBeans changes frequently, please check updates frequently too: "Tool"->"Plugins"->"Reload Category" (please update all available modules, including those not relating to Scala). If things are broken, re-download a new nightly build and try ...

Scala on NetBeans Modules on Update Center Now

>>> Updated Feb 2, 2008:

There is an issue in auto-generated Scala project's ant build-impl.xml file, which get the scala.home property wrongly set. I just fixed it, please wait the new Scala Project module to be available on Update Center, which version should be 1.2.1.

======

I've added Scala modules to nightly build cluster, so, you can get/update these modules via NetBeans plugins update center from today.

Since these modules are experimental, I recommend you to download a nightly build version NetBeans before get these modules, you can get the latest nightly build NetBeans from:  http://bits.netbeans.org/download/trunk/nightly/latest/

To get the Scala modules, open menu: [Tools]->[Plugins]</b>, check [Setting] to ensure "Last Development Build" is in the list of Update Centers, which with Url:

 http://deadlock.netbeans.org/hudson/job/javadoc-nbms/lastSuccessfulBuild/artifact/nbbuild/nbms/updates.xml.gz

Then in the [Available Plugins] tab, you can find the "Scala" category, which currently contains 3 modules: Scala Console, Scala Editing, Scala Project (you may need to click [Reload Category]). Select them and click [Install]

Notes:

  • Don't forget to set your <b>SCALA_HOME</b> environment first, and append -J-Dscala.home=scalahomepath (for Windows users, try to append -J-Dscala.home=%SCALA_HOME%) to the end of "netbeans_default_options" in your netbeans.conf file, where, scalahomepath is your Scala home's absolute path. For example: /Users/dcaoyuan/apps/scala/share/scala/ (which contains sub-directory: bin, lib etc). netbeans.conf is located at "pathToNetBeansInstallationDirectory/etc", in Mac OSX, it could be:
    • /Applications/NetBeans/NetBeans\ 6.0.app/Contents/Resources/NetBeans/etc, or ~/SomePath/netbeans/etc
  • The Scala Project supporting has been rewritten, so if you have Scala projects created by previous Scala plugin, you should recreate new Scala project, and copy source files to this new created project's src directory.
  • Debugger module is not released yet.

Scala for NetBeans Screenshot#7: Working on Debug II

So, I've found the cause of that can't add breakpoints on context of object. By setting the enclosing class name of object as object's name + "$", I can now add breakpoints upon object.

To reach here, I have to write an EditorContextImpl? for Scala, which will get all compilation information from Scala semantic analyzer. But, to get all debugging features working, I have still to process a bit more conditions. After that, I'll release these modules for public trying.

Click on the picture to enlarge it

ScalaEditor_080130.png

Scala for NetBeans Screenshot#6: Working on Debug

=== Updated Jan 30, 08: ==>
The cause of adding breakpoints to object context is relative to complilationInfo's CompilationUnit in java.source module, that the ComplilationInfo is set by JavaSource#moveToPhase after call Iterable trees = currentInfo.getJavacTask().parse(new JavaFileObject[] {currentInfo.jfo}), thus the returning CompilationUnitTree is generated by sun's java parser tool, it of course can not be parsed properly with "object" keyword.

To resolve it, I have to rewrite a scala.debug.projects module, which will be derived from debugger.jpda.projects module
===

=== Updated Jan 26, 08: ==>
Yes, I've got a better Scala project support, not only debug works on pure Scala or Scala+Java project, but also got an UI for defining Main class of the application. Actually, Scala project support is now almost same as J2SE project on NetBeans. The only problem is, if you add a breakpoint on statements of a Scala object, since the org.netbeans.api.java.source.TreeUtilities#scopeFor will return a scope of "COMPILATION_UNIT", which will return null when is called by scope.getEnclosingClass() in org.netbeans.modules.debugger.jpda.projects.EditorContextImpl#getClassName, so the breakpoint will be invalid. To get this fixed, EditorContextImpl needs to be patched.

According to Scala FAQ, Scala object gets compiled to 2 classes, for example:

object HelloObj {
  val f = "one field"
  val x = 3
}

We will get:

public final class HelloObj extends java.lang.Object{
    public static final int $tag();
    public static final int x();
    public static final java.lang.String f();
}
public final class HelloObj$ extends java.lang.Object implements scala.ScalaObject{
    public static final HelloObj$ MODULE$;
    public static {};
    public HelloObj$();
    public int x();
    public java.lang.String f();
    public int $tag();
}

Writing “HelloObj.x()” in Scala is the same as writing “HelloObj$.MODULE$.x()”, but the former should probably be preferred for readability.

The problem here is, when you add breakpoint at val x = 3 in the source file, what will be the breakpoint's scope in these two compiled classes (which one?), why scope.getClassName() will return null? anyone know what's the magic here?
===

I'm working on debug feature for Scala on NetBeans. I copied a smallest set of classes from Java Debugger module, and hacked something to make debug working on a mixed project (a Java project with Scala source files). By default, NetBeans IDE will step through Scala source files that are called (see Greetjan's article: Stepping through Groovy in NetBeans IDE ), but you can't add breakpoints directly on Scala source file. So the first thing that I should work on is let the user can add/remove breakpoints in Scala Editor, and get NetBeans' debugger stopping at these breakpoints. After a full day hacking, I got this working.

To got mouse click on adding/removing breakpoints working, you can implement and register a MIMEType related "GlyphGutterActions" in layer.xml, so, org.netbeans.modules.editor.impl.GlyphGutterActionsProvider#getGlyphGutterActions(String mimeType) can lookup this action. GlyphGutterActionsProvider#getGlyphGutterActions will be invoked by org.netbeans.editor.GlyphGutter#GutterMouseListener#mouseClicked, that's it.

The screenshot shows a mixed Java/Scala project, the example code was copied from Fred Janon's blog, it's a pretty simple example. I added a breakpoint on Dog.scala's val sound="Woof woof!" and another breakpoint at function talk()'s println(hi + sound), then invoke debug, the execution stopped at these two breakpoints as I expected, and you can see the context, for example, the values of variable "sound" and "hi" on debugging windows (at the bottom)

Of course, you can step into, step to cursor line too.

I need to get the debug feature working on a pure Scala project, I have not tried yet, but it seems I need to get the Scala project module better before that.

Click on the picture to enlarge it

nn

Scala Supporting for NetBeans Updated#2 - 20080112

Warning:

Scala for NetBeans is still under development. All these releases are experimental, they may be unstable yet.

Change Log:

  • Can be installed on NetBeans 6.0
  • Fixed some syntax broken
  • Various bugs fixes
Features:

  • Syntax checking, highlighting, code folding, navigator, basic indent
  • Basic completion, In-place refactoring, occurrences marks
  • Interactive Scala shell. [Windows] -> [Interactive Scala Shell]
  • Basic Scala project management with file locator for compile Errors
Notes:

  • Requires NetBeans 6.0 or newest NetBeans Nightly build (get from: http://bits.netbeans.org/download/trunk/nightly/latest/)
  • Don't forget to set your SCALA_HOME environment first, or append "-J-Dscala.home=scalahomepath" to the end of "netbeans_default_options" in your netbeans.conf file, where, "scalahomepath" is your Scala home's absolute path. For example: /Users/dcaoyuan/apps/scala/share/scala/ (For Windows users, it's better to set both of them).
    The netbeans.conf is located at "pathToNetBeansInstallationDirectory/etc", in Mac OSX, it could be:
    /Applications/NetBeans/NetBeans\ 6.0.app/Contents/Resources/NetBeans/etc

Known Issues:

  • Do not write old-style ForComprehension "for (val i <- ...)", instead, use "for(i <- ...)". Please see Scala Spec 2.6.0+
  • Do not put infix/postfix operator at the beginning of new line even in a parenthesis expression
  • When "<" is an operator, put a space after "<" to identify it from a xml element

Download:

  • http://sourceforge.net/project/showfiles.php?group_id=192439&package_id=256544

Installation:

  1. Upzip to some where, there will be several *.nbm files
  2. Run NetBeans, install these *.nbm files via [Tools] -> [Plugins] -> "Downloaded"

Scala Supporting for NetBeans Updated#1 - 20080110

Features:

  • Syntax checking, highlighting, code folding, navigator, basic indent
  • Basic completion, In-place refactoring, occurrences marks
  • Interactive Scala shell. [Windows] -> [Interactive Scala Shell]
  • Basic Scala project management with file locator for compile Errors
Notes:

  • Requires newest NetBeans Nightly build (get from: http://bits.netbeans.org/download/trunk/nightly/latest/)
  • Don't forget to set your SCALA_HOME environment first, or append "-J-Dscala.home=scalahomepath" to the end of "netbeans_default_options" in your netbeans.conf file, where, "scalahomepath" is your Scala home's absolute path. For example: /Users/dcaoyuan/apps/scala/share/scala/
    The netbeans.conf is located at "pathToNetBeansInstallationDirectory/etc", in Mac OSX, it could be:
    /Applications/NetBeans/NetBeans\ 6.0.app/Contents/Resources/NetBeans/etc

Known Issues:

  • Do not write old-style ForComprehension "for (val i <- ...)", instead, use "for(i <- ...)". Please see Scala Spec 2.6.0+
  • Do not put infix/postfix operator at the beginning of new line even in a parenthesis expression
  • When "<" is an operator, put a space after "<" to identify it from a xml element

Download:

  • http://sourceforge.net/project/showfiles.php?group_id=192439&package_id=256544

Installation:

  1. Upzip to some where, there will be several *.nbm files
  2. Run NetBeans, install these *.nbm files via [Tools] -> [Plugins] -> "Downloaded"

Scala for NetBeans Screenshot#5: In-Place Refactoring

I've got Scala's semantic analysis more tightly integrated into NetBeans' Schliemann module. The first benefit is In-Place Refactoring (Instant Rename).

On the following screenshots, first, you put caret on a variable name ("scope", for example), the occurrences of this variable will be hi-lighted. Then, press key Ctrl+R, the highlighting color will be changed with meaning of ready for rename.

Just type or modify the name to new one, then press Enter. The occurrences of "scope" will be changed to new name, for example: "scope1".

nn

nn

And, YES, the Erlang for NetBeans will also get this benefit.

The Year That Was, The Year That Will Be

AIOTrade

It reached 36,000 download. I use AIOTrade for my trading, via my Neural Network prediction. For 2007, I invested 1 and got almost 300%, not bad. For trading in 2008, I'm now on opened position, let me see what another year will be. For AIOTrade itself in 2008, you may have guessed (or not), I'm planning to rewrite AIOTrade in Scala.

ErlyBird

ErlyBird is now 0.15.2. It reached 2,500 downlaod. For 2008, I hope to integrate more features into NetBeans Schliemann project. For 2008, Erlang as a programming language will not be the choice for AIOTrade, I may tell why some day.

Scala for NetBeans

I'm a such lazy man, that I have to write some helpful tools before I do something. That's why, before trade, I had to write AIOTrade, before rewrite AIOTrade (Plan A - in Erlang, now dropped), I have to write ErlyBird, before rewrite AIOTrade (Plan B - in Scala) I have to write Scala for NetBeans. That's the cause-chain. Thanks God, I do not need to invent another language, there has been Scala.

Well, Scala looks very powerful and interesting, it's what in my eyes, a clean mix of Java + Ruby + Erlang, or, the JRE (Just Running Environment).

But it's so powerful in syntax, that is so difficult for a truly IDE supporting, that is so important to have an IDE as assistant (which will help writing Scala a lot). And, since I'm going to build things upon NetBeans as NetBeans' modules, the IDE of course has to be based upon NetBeans. I really hope there had been a good Scala supporting for NetBeans, but for 2007, there was none. So, how about 2008?

Happy New Year, everybody.

Happy New Year, especially to Beijing, the 2008 Olympic Game, although Life Is Always Elsewhere.

Scala for NetBeans Screenshot#4: Basic Completion

Scala for NetBeans now supports basic completion for keywords, functions of Predef, local var/val/function etc.

The screenshot shows the popup with candidate items when you typed "s" and pressed Ctrl+Space, including functions: scalProd(), sum() and val: scope in context of for.scala

Click on the picture to enlarge it

nn

Scala for NetBeans Screenshot#3: Preliminary Marking Occurrences and Goto Declaration

When you put the caret on a var, val, or function id, for example, "userName" on the below screenshot, the occurrences of this var/val/function will be highlighting, and, on the right side bar, there are colored indicators telling you where are these occurrences.

You can also go to declaration of var/val/function too (only in same source file currently).

What does this mean? It means I've begun to do some semantic analysis on parsed AST.

Click on the picture to enlarge it

nn

First Experimental Scala Supporting for NetBeans is Available

>>> Updated Dec 23: There is an updated Editing module at:
http://sourceforge.net/project/showfiles.php?group_id=192439&package_id=256544&release_id=563760

CHANGELOG:

  • "{" of class/object/trait declarations can be put on new line
  • Some broken syntax fixed.

========

Features:

  • Syntax checking, highlighting, code folding, navigator, basic indent
  • Interactive Scala shell. [Windows] -> [Interactive Scala Shell]
  • Basic Scala project management with file locator for compile Errors
Notes:

  • Requires NetBeans 6.0 Release
  • Don't forget to set your SCALA_HOME environment first, or append "-J-Dscala.home=scalahomepath" to the end of "netbeans_default_options" in your netbeans.conf file, where, "scalahomepath" is your Scala home's absolute path. For example: /Users/dcaoyuan/apps/scala/share/scala/
    The netbeans.conf is located at "pathToNetBeansInstallationDirectory/etc", in Mac OSX, it could be:
    /Applications/NetBeans/NetBeans 6.0.app/Contents/Resources/NetBeans/etc

Know Issues:

  • Embedded /* */ comment is not supported yet.
  • Do not write old-style ForComprehension "for (val i <- ...)", instead, use "for(i <- ...)". Please see Scala Spec 2.6.0+
  • Do not put infix/postfix operator at the beginning of new line even in a parenthesis expression
  • When "<" is an operator, put a space after "<" to identify it from a xml element

Download:

  • http://sourceforge.net/project/showfiles.php?group_id=192439&package_id=256544

Installation:

  1. Upzip to some where, there will be several *.nbm files
  2. Run NetBeans, install these *.nbm files via [Tools] -> [Plugins] -> "Downloaded"

Scala for Netbeans Screenshot#2: Scala Console and File Locator for Compile Errors

The interactive Scala shell console works now, with 'up'/'down' key for command history.

And as ant based Scala project, the project building just works fine, in the building output window, you can click on the compiler error message to jump to the corresponding location of source file.

You may have also noticed, if an id is used as operator, it's highlighting in special color.

The case clauses are listed in the outline window.

Click on the picture to enlarge it

nn

ErlyBird 0.15.2 Released - An Erlang IDE based on NetBeans

I'm pleased to announce ErlyBird 0.15.2, an Erlang IDE based on NetBeans. This is an important feature release in size of 17.9M.

CHANGELOG:

  • Supported OTP/Erlang R12B new syntax.
  • A new Emacs standard color theme.
  • Fixed some formatter bugs.
  • Better syntax error message.
  • Various bugs fixes.

To switch color theme, open [Tools]->[Options], click on 'Fonts & Colors', choose 'Profile' drop-down box.

Java JRE 5.0+ is required.

To download, please go to: http://sourceforge.net/project/showfiles.php?group_id=192439

To install:

  1. Unzip erlybird-bin-0.15.2-ide.zip to somewhere.
  2. Make sure 'erl.exe' or 'erl' is under your environment path
  3. For Windows user, execute 'bin/erlybird.exe'. For *nix user, 'bin/erlybird'.
  4. Check/set your OTP path. From [Tools]->[Options], click on 'Erlang', then 'Erlang Installation' tab, fill in the full path of your 'erl.exe' or 'erl' file. For instance: "C:/erl/bin/erl.exe"
  5. The default -Xmx option for jvm is set to 256M, ErlyBird now works good with less memory, such as -Xmx128M. If you want to increase/decrease it, please open the config file that is located at etc/erlybird.conf, set -J-Xmx of 'default_options'.

When run ErlyBird first time, the OTP libs will be indexed. The indexing time varies from 10 to 30 minutes deponding on your computer.

Notice: If you have previous version ErlyBird installed, it's recommended to delete the old cache files which are located at:

  • *nix: "${HOME}/.erlybird/dev"
  • mac os x: "${HOME}/Library/Application Support/erlybird/dev"
  • windows: "C:\Documents and Settings\yourusername\.erlybird\dev" or some where

The status of ErlyBird is still Alpha, feedbacks and bug reports are welcome.

Scala Support for NetBeans Screenshot#1: Syntax Highlighting and Scala Project

Scala editor module has been integrated with existing Scala Project module. The xml syntax is almost supported. There are still a little bit complex syntax not be supported yet.

I hope a downloadable Scala modules package can be ready in one week, so you can get it from the NetBeans update center.

BTW, new version of ErlyBird is not ready yet, I'm waiting for fixing of some issues in NetBeans' Generic Languages Framework module.

Screen snapshot: (Click to enlarge)

ScalaEditor_071212.png

Scala Editor for NetBeans

>>> Updated Dec 10: I've been granted developer role for netbeans' languages project, and committed the basic editing support code.
========

I wrote another NetBeans language supporting module, A Scala Editor. I've got most syntax working with syntax checking, highlighting, fold, navigator etc, except the xml syntax.

Since Scala is a newline aware language, the LL(k) grammar definitions is a challenge, I spent all my 2 days of this weekend to reach here.

There are a lot of works left for a full featured editor, after that, I'll release it to public, and contribute to NetBeans community.

Here's a screen snapshot: (Click to enlarge)

nn

ErlyBird Is Ready for R12B

OTP/Erlang R12B is coming soon, which will support Binary Comprehension and -spec, -type attributes. I've updated ErlyBird to support these new syntax, and, with a new Emacs Standard color theme.

The new version of ErlyBird will be available around the releasing date of OTP/Erlang R12B and NetBeans 6.0

ErlyBird 0.15.1 Released - An Erlang IDE based on NetBeans

I'm pleased to announce ErlyBird 0.15.1, an Erlang IDE based on NetBeans. This is a performance improvement release. This release will only provide all-in-one IDE package, which is in size of 18.3M.

CHANGELOG:

  • Performance improvement.
  • Integrated with NetBeans' Common Scripting Framework. Thanks Tor.
  • Fix a bug related to occurrences mark on built-in functions.
  • Fix bug of wrong formatting multiple-lines string.
  • Supports "-module(x.y.z)" syntax.
  • Various bugs fixes.

Java JRE 5.0+ is required.

To download, please go to: http://sourceforge.net/project/showfiles.php?group_id=192439

To install:

  1. Unzip erlybird-bin-0.15.1-ide.zip to somewhere.
  2. Make sure 'erl.exe' or 'erl' is under your environment path
  3. For Windows user, execute 'bin/erlybird.exe'. For *nix user, 'bin/erlybird'.
  4. Check/set your OTP path. From [Tools]->[Options], click on 'Erlang', then 'Erlang Installation' tab, fill in the full path of your 'erl.exe' or 'erl' file. For instance: "C:/erl/bin/erl.exe"
  5. The default -Xmx option for jvm is set to 256M, ErlyBird now works good with less memory, such as -Xmx128M. If you want to increase/decrease it, please open the config file that is located at etc/erlybird.conf, set -J-Xmx of 'default_options'.

When run ErlyBird first time, the OTP libs will be indexed. The indexing time varies from 10 to 30 minutes deponding on your computer.

Notice: If you have previous version of ErlyBird 0.12.0+ installed, you can keep your old cache files, otherwise, please delete the old cache files which are located at:

  • *nix: "${HOME}/.erlybird/dev"
  • mac os x: "${HOME}/Library/Application Support/erlybird/dev"
  • windows: "C:\Documents and Settings\yourusername\.erlybird\dev" or some where

The status of ErlyBird is still Alpha, feedbacks and bug reports are welcome.

ErlyBird 0.15.0 Released - An Erlang IDE based on NetBeans

Updated(Sep 23): Known issues:

  • Formmater: String literal that is broken to multiple lines will be reformatted, this should be fixed. (Done in trunk)
  • 'Run project' does not work yet.
  • When more than one projects are opened concurrently, 'Go to declaration' for remote function may not work. You can close other projects when encountered.

I'm pleased to announce ErlyBird 0.15.0, an Erlang IDE based on NetBeans. This is a major features release. This release will only provide all-in-one IDE package, which is in size of 17.6M.

CHANGELOG:

  • Pretty formatter (Ctrl+Shift+F).
  • Variables and functions occurrences mark.
  • Better brace matching highlighting, such as for 'try-catch-end', 'if-end' etc.
  • "-import" syntax now works in all cases, that means RabbitMQ's code will be parsed correctly.
  • Various bugs fixes.

As NetBeans 6.0 beta1 was just released, I hope ErlyBird has got more stable also.

Java JRE 5.0+ is required.

To download, please go to: http://sourceforge.net/project/showfiles.php?group_id=192439

To install:

  1. Unzip erlybird-bin-0.15.0-ide.zip to somewhere.
  2. Make sure 'erl.exe' or 'erl' is under your environment path
  3. For Windows user, execute 'bin/erlybird.exe'. For *nix user, 'bin/erlybird'.
  4. Check/set your OTP path. From [Tools]->[Options], click on 'Erlang', then 'Erlang Installation' tab, fill in the full path of your 'erl.exe' or 'erl' file. For instance: "C:/erl/bin/erl.exe"
  5. The default -Xmx option for jvm is set to 256M, ErlyBird now works good with less memory, such as -Xmx128M. If you want to increase/decrease it, please open the config file that is located at etc/erlybird.conf, set -J-Xmx of 'default_options'.

When run ErlyBird first time, the OTP libs will be indexed. The indexing time varies from 10 to 30 minutes deponding on your computer.

Notice: If you have previous version of ErlyBird 0.12.0+ installed, you can keep your old cache files, otherwise, please delete the old cache files which are located at:

  • *nix: "${HOME}/.erlybird/dev"
  • mac os x: "${HOME}/Library/Application Support/erlybird/dev"
  • windows: "C:\Documents and Settings\yourusername\.erlybird\dev" or some where

The status of ErlyBird is still Alpha, feedbacks and bug reports are welcome.

ErlyBird Screenshot: Brace Matching and Mark Occurences

I patched NetBeans' Generic Languages Framework: Schliemann, and got Brace Matching and Mark Occurences working perfectly on ErlyBird now. As shown on the screenshot, when you put the caret on "end" or "case", the matched "case" or "end' will be highlighing. When you put the caret on a variable, for example, "State", the occurrences of the variable will be highlighting, and, on the right side bar, there are colored indicators telling you where are these occurrences. The occurrences marking also works for function names.

The coming release also fixed some bugs, such as the "import" syntax now works in all cases, that means RabbitMQ's code will be parsed correctly.

Click on the picture to enlarge it

nn

ErlyBird 0.12.1 released - Erlang IDE based on NetBeans

I'm pleased to announce ErlyBird 0.12.1, an Erlang IDE based on NetBeans.

This is a performance improvement release. This release will only provide all-in-one IDE package, which is in size of 17.3M.

By refining the LL(k) definition of Erlang syntax, I've got ErlyBird parsing performance improved a lot, for example, the time of indexing whole OTP libs is cut to half now.

And this is the first time, ErlyBird works smoothly enough in my compter. I'm with full confidence on Generic Languages Framework of NetBeans now.

Java JRE 5.0+ is requested.

To download, please go to: http://sourceforge.net/project/showfiles.php?group_id=192439

To install:

  1. Unzip erlybird-bin-0.12.1-ide.zip to somewhere. For Windows user, execute 'bin/erlybird.exe'. For *nix user, 'bin/erlybird'.
  2. Check/set your OTP path. From [Tools]->[Options], click on 'Miscellanous', then expand 'Erlang Installation', fill in the full path of your 'erl.exe' or 'erl' file. For instance: "C:/erl/bin/erl.exe"
  3. The default -Xmx option for jvm is set to 256M, if you want to increase it, please open the config file that is located at etc/erlybird.conf, set -J-Xmx of 'default_options'.

When you run ErlyBird first time, the OTP libs will be indexed. The indexing time varies from 10 to 30 minutes deponding on your computer.

Notice: If you have previous version of ErlyBird installed, please delete the old cache files which are located at:

  • *nix: "${HOME}/.erlybird/dev"
  • mac os x: "${HOME}/Library/Application Support/erlybird/dev"
  • windows: "C:\Documents and Settings\yourusername\.erlybird\dev" or some where

The status of ErlyBird is still Alpha, feedback and bug reports are welcome.

CHANGELOG:

  • Performance improvement

Vi Module Meets ErlyBird

There have been several Erlang development tools: Erlang module for Emacs, for vim, and Erlide for Eclipse. Why I write another Erlang IDE based on NetBeans?

Erlang for Emacs runs smoothly on my computer, but the distel module can not communicate with Erlang node on my Windows XP, that means I can not have the auto-completion, go to declaration features working; Erlang for vim is not a complete IDE yet; Erlide hangs on my Windows XP too. So I write ErlyBird.

But I'm actually a vi fun, so I just download and install the excellent jVi module to ErlyBird, which is a fully functional vi environment with good performance. There is an article talking about vi integration with NetBeans, which can also be applied to ErlyBird.

I'm now with fun with Vi on ErlyBird on my daily job.

The biggest issue for ErlyBird currently is the rendering performance, which causes performance slowing down if you run ErlyBird a while. I'm not sure if this issue depends on Generic Language Framework module of NetBeans. After I get the new laptop which with 2G memory next week, I may do some profile analysis.

I've also written some code to talk with Erlang Node from ErlyBird, everything looks smooth too.

I'll fly to San Francisco next week, to meet my new and old friends there.

It seems that this has been a world you should mix Vi/Netbeans, Java/Erlang, Beijing/Vancouver/San Francisco, whatever, together? A dynamical, colorful, multi-culture world, you have to look for the truths carefully, continually.

ErlyBird 0.11.2 released - An Erlang IDE based on NetBeans

I'm pleased to announce ErlyBird 0.11.2, an Erlang IDE based on NetBeans.

This is a bug-fix, stabilization release. Since I tightly modified GSF/GLF modules of NetBeans, this release will only provide all-in-one IDE package, which is in size of 14.8M.

To download, please go to: http://sourceforge.net/project/showfiles.php?group_id=192439

To install:

  1. Unzip erlybird-bin-0.11.2.zip to somewhere. For Windows user, executee 'bin/erlybird.exe'. For *nix user, 'bin/erlybird'.
  2. Check/set your OTP path. From [Tools]->[Options], click on 'Miscellanous', then expand 'Erlang Installation', fill in the full path of your 'erl.exe' or 'erl' file. For instance: "C:/erl/bin/erl.exe"
  3. The default -Xmx option for jvm is set to 256M, if you want to increase, open the config file of ErlyBird that is located at etc/erlybird.conf, set -J-Xmx in line of 'default_options'

The status of ErlyBird is still Alpha, feedback and bug reports are welcome.

CHANGELOG:

  • Indexing will skip too big files according to the max memeory. This avoids ErlyBird to hang when indexing.
  • If erl/erl.exe is under the environment path, ErlyBird will try to set Erlang Installation path automatically.
  • Including function args in completion suggestion.
  • Various bugs fixes especially for stabilization.

ErlyBird Screenshot: Including Args in Completion Suggestion

As a newbie to Erlang, I'm not familiar with those OTP module/functions, I have to go back to see the docs again and again. At least, now ErlyBird will suggest me the arguments of each function now.

Click on the picture to enlarge it

nn

Other progress:

  • If erl/erl.exe is under the environment path, ErlyBird will try to set Erlang Installation path automatically.
  • Indexing engine is now based on Lucene, and is integrated with NetBeans' Generic Scripting Framework 100%.
  • As I'm now using ErlyBird for my daily works, and have written more Erlang code, I fixed some bugs that cause ErlyBird unstable.

To do list in the near future:

  • Integrate with Erlware to provide better project manager.
  • Go to declaration of macro
  • Completion for fields of record
  • Highlighting unbound variables and unused variables
  • Basic re-factor features: rename local variable name, un-exported function name etc.
  • Manage Erlyweb project
  • Support syntax of Erlyweb template: *.et files
  • Stabilize, Stabilize

And I hope language engine of NetBeans will be optimized soon, since it eats too much memory and needs performance improvement.

Write an IDE in One Month - ErlyBird 0.11.0 Released

Updated Apr 24: The indexing feature is based on Lucene indexing from Common Scripting Framework now. But there won't be new release soon, since Generic Language Framework changes rapidly.

Updated Apr 21: There are several source files under \lib\megaco-3.5.3\src\text, which are with size > 300k, if you can not pass indexing procedure (ErlyBird hangs), please rename them something that are not end with ".erl" or ".hrl".

Updated Apr 20: Due to a severe bug that prevents setting Erlang Installation path, I've re-pack a new release 0.11.1 that fixed it. Don't forget to set the Erlang Installation path to full path of erl.exe or erl file, for instance, "C:\erl\bin\erl.exe".

Updated Apr 20: There is a bug, if your OTP is not installed as C:\erl, you can not set Erlang Installation. I'll update the bin package.

Updated Apr 20: erlybird-bin-0.11.0-ide.zip has been uploaded to sourceforge.net.

I'm newbie to IDE, to Erlang, to compilers principles. But, based on the works from NetBeans's guys, I wrote an almost complete Erlang IDE in one month.

With features:

  • Syntax checking.
  • Syntax highlighting.
  • Functions navigator.
  • Code-folding.
  • Indentation.
  • Completion for built-in/remote function call. (Ctrl+Space for suggestion) (new)
  • Go to declaration for remote function call (press down Ctrl and click on the function name). (new)
  • Project management - create/manage/build project tree. (new)
  • Compilation error locate. (new)
  • Auto indexing OTP libs and project sources. (new)
  • Interactive Erlang Console. (new)

And with known Issues:

  • ErlyBird is memory eager so far, it needs at least -Xmx256m set, 500m or more is recommended for big source files. Check the config file of NetBeans that is located at etc/netbeans.conf, make sure you've set -J-Xmx256m in line of 'netbeans_default_options'
  • Run project button does not work yet, if you press run project button, will show an interactive erlang shell only.
  • When indexing OTP libs, there may be Exceptions pop-ups, which indicate out of heap space, you can just ignore it.
  • Do not open/go to declaration too big source file in ErlyBird, this will also cause out of heap space.

And lacking features:

  • Debugging
  • ...

The parsing and editing features is based on Generic Language Framework.

The project management and indexing features is based on Common Scripting Framework.

To integrate editing feature, which is from Generic Language Framework, with Comman Scripting Framework, I had to lightly modify the Common Scripting Framework source code, the modified files are available from svn trunk folder: gsf-diff-ref.

And as I'm not yet familiar with Lucene index engine, which is used by Common Scripting Framework, I just wrote a sql db index engine and plug-in it to Common Scripting Framework. I will rewrite it based on LuceneIndex lately.

To download, please go to:

http://sourceforge.net/project/showfiles.php?group_id=192439

There are two installation options now, you can choose one of them:

The first one: A pre-packed NBMs kit: erlybird-bin-0.11.1-kit-nbms.zip(about 2.8M). To install:
Downloaded NetBeans IDE 6.0 M8+ via:
http://www.netbeans.info/downloads/dev.php
Select 'Milestone' in 'Build Type'.
After NetBeans IDE installed, unzip erlybird-bin-0.11.1-kit-nbms.zip first, then:

  1. From menu: Tools -> Update Center
  2. In the "Select Location of Modules" pane, click "Install Manually Downloaded Modules(.nbm Files)", then "Next"
  3. Click [Add...] button, go to the path to select all *.nbm files.
  4. Following the instructions to install updated modules.
  5. Restart NetBeans.
  6. Set your OTP path. From [Tools]->[Options], click on 'Miscellanous', then expand 'Erlang Installation', fill in the full path of your 'erl.exe' or 'erl' file, for instance: "C:/erl/bin/erl.exe"

The second one: A standalone ErlyBird IDE: erlybird-bin-0.11.1-ide.zip(about 18M). Notice: Please wait for me to upload it to sf.net :-) It does not need NetBeans IDE. To install:

  1. Just unzip it to somewhere, then execute 'bin/erlybird.exe' for windows, 'bin/erlybird' for *nix.
  2. Set your OTP path. From [Tools]->[Options], click on 'Miscellanous', then expand 'Erlang Installation', fill in the full path of your 'erl.exe' or 'erl' file, for instance: "C:/erl/bin/erl.exe"

If you are new to NetBeans, there are some docs for user:

http://www.netbeans.org/kb/55/quickstart.html
http://www.netbeans.org/kb/55/using-netbeans/index.html

It may not be stable yet, feedback and bug reports are welcome.

ErlyBird Screenshot: Completion/Go to Declaration for remote function call

ErlyBird now support completion for remote function call. And, if you click on the name of a remote function call, ErlyBird will open the remote module source file, and jump to the position of declaration. (Only OTP modules are supported currently)

The screenshot shows the popup with candidate functions of module "file", when you typing "file:"

I'll release a new version in one week.

Click on the picture to enlarge it

nn

ErlyBird Screenshot: Erlang Console and File Locator for Compile Errors

Updated Apr 11: copy/paste can be done via Ctrl+C/Ctrl+V.

The Erlang Shell console finally works on NetBeans, it works as same as on the shell/dos environment with historical commands feature. It still lacks copy/paste feature.

And ErlyBird supported Emakefile based project build. In the output window, you can click on the compiler error message to jump to the corresponding location of source file.

Click on the picture to enlarge it

nn

Erlang Project Support and Code Completion in ErlyBird

I've got the initial Erlang project management supported in ErlyBird, where the Erlang project tree can be newly created and managed by NetBeans. The code is ported from Tor's work for Ruby in NetBeans, and I'll try to sync my work with his work.

The Erlang interactive console does not truly work yet:-) but it's the first priority task.

Click on the picture to enlarge it

nn

Go to Declaration of Function call and Var in Erlang Editor for Netbeans

I've got "Go to declaration of function call and var" if the declarations are in the same module file, and "Highlighting for function call/function arguments" working.

To go to the declaration of function call or var, just press down "Ctrl", and put cursor on the function call or var name's position, then click on it. The editor will jump to the source position of declaration.

But to get cross-module "Go to declaration of function call" working, I may need much more works to do.

BTW, the Erlang project management is also under developing. Before this feature is released, the only method to create a managed project in NetBeans is create a Java project tree and use it.

Click on the picture to enlarge it

nn

Three Interviews with Language Programmers for NetBeans

Geertjan from NetBeans took three interviews with language programmers for NetBeans, one of them is me, where I talked about ErlyBird - Erlang Editor for NetBeans and AIOTrade (formly HumaiTrader) which are all based on NetBeans Platform.

About AIOTrade, in the interview, I said:

"I'm going to split AIOTrade into a client/server application. The server-side will act as a streaming quote data feed server and be responsible for delivering transaction orders to brokers in soft real-time, written in Erlang. The client-side charting and UI features will remain in Java, where Java is strong. With the Jinterface APIs provided by Erlang, Java is easily able to talk with the Erlang server."

Erlang Editor for NetBeans - ErlyBird 0.10.1 released

Update - Mar 29,2007: If you got exception: java.lang.reflect.InvocationTargetException when try completion, please check the version number of your "Generic Languages Framework" module (Tools -> Module Manager -> Language Support), if the version number is less than 1.70, you can go to http://sourceforge.net/projects/erlybird to download and update to the newly built org-netbeans-modules-languages.nbm

I'm pleased to announce ErlyBird 0.10.1, an Erlang Editor Module for NetBeans has been released.

Current features:

  • Syntax checking;
  • Syntax highlighting;
  • Functions navigator;
  • Code-folding;
  • Indentation;
  • Built-in function completion.

You can download ErlyBird from http://sourceforge.net/projects/erlybird

ErlyBird needs NetBeans IDE 6.0 M7+, which can be downloaded via:
http://www.netbeans.info/downloads/dev.php page
select Q-Build in 'Build Type'.

After NetBeans IDE installed, go to Tools->Update Center, fetch the "Generic Language Framework" module from Category "Languages Support"

To install ErlyBird module, unzip the binary package first, then:

  1. From menu: Tools -> Update Center
  2. In the "Select Location of Modules" pane, click "Install Manually Downloaded Modules(.nbm Files)", then "Next"
  3. Click [Add...] button, go to the path to select the unzip .nbm file.
  4. Following the instructions to install updated modules.
  5. Restart NetBeans.

It may not be stable yet, feedback and bug reports are welcome.

Erlang Editor Support Based on NetBeans' Generic Language Framework

I did some work to get Erlang editor supported on NetBeans. As the Schliemann project (Generic Language Framework) is still under developing, I just got simple syntax coloring, indentation, code folding etc. working. I'll contribute it to NetBeans Community when it's stable enough. Here is a snapshot:

Click on the picture to enlarge it

nn

Ruby IDE for NetBeans Almost Useful

As NetBeans IDE 6.0M7 released, I tried the Ruby module for it, and it's almost useful now.

To get and install,

1. Downloand NetBeans IDE 6.0M7 from:
http://www.netbeans.info/downloads/dev.php
Select 'Q-Build' and download the newest M7

2. Update Ruby modules:
1) [Tools] -> [Update Center]
2) Select Ruby folder as you wanted (9 files will be selected)
3) Following the instructions.

3. Set your Ruby environment:
As the default installation will use JRuby, if you want to use c-ruby, go to
1) [Tools]->[Options]->Miscellaneous->Ruby Installation
2) Change all ruby tools to yours

4. Now setup your first Ruby on Rails Application:
1) [File]->[New Project]->Ruby->Ruby on Rails Application
2) If you have an existed project, copy and override to the new created project tree.

Want to take a look at the snapshot? here it is:
NetBeans + Ruby = True

That's all. Have fun with great NetBeans.

Notice: If you are using c-ruby, don't try to run project via NetBeans' "run main project" button, which may change your environment temporarily.

It Will Be My First Attendance at NetBeans Day, Seattle

I will be there, Sun Tech Days, Seattle, Sep 6, 2006. As I'm now in Vancouver, it's about 2 or 3 hours trip to Seattle.

I'm glad to have a chance to meet those great guys who develop NetBeans IDE and Platform. As you know, the AIOTrade (formerly Humai Trader) is built on NetBeans Platform using NetBeans IDE.

And, I've committed the re-packed source code to SVN repository on sourceforge.net, and am doing cleanup on the neural network code, hope to commit the code in one week.

For the neural network module, there should be a lot of UI works still needed to be done, I've been beginning to hack the Visual Library API of NetBeans, and hope to apply these great works on visual neural network definition.

How to add a dropdown menu to toolbar in NetBeans Platform

To add a dropdown menu to toolbar, you should extends CallableSystemAction and override method: public Component getToolbarPresenter()

Dropdown menu should be banded with a JToggleButton, when JToggleButton is pressed down, show a JPopupMenu on JToggleButton.

Example:

public class PickDrawingLineAction extends CallableSystemAction {
    private static JToggleButton toggleButton;
    private static ButtonGroup buttonGroup;
    private static JPopupMenu popup;
    private MyMenuItemListener menuItemListener;
    
    List<AbstractHandledChart> handledCharts;
    
    public PickDrawingLineAction() {
    }
    
    
    public void performAction() {
        java.awt.EventQueue.invokeLater(new Runnable() {
            public void run() {
                toggleButton.setSelected(true);
            }
        });
        
    }
    
    public String getName() {
        return "Pick Drawing Line";
    }
    
    public HelpCtx getHelpCtx() {
        return HelpCtx.DEFAULT_HELP;
    }
    
    protected boolean asynchronous() {
        return false;
    }
    
    public Component getToolbarPresenter() {
        Image iconImage = Utilities.loadImage(
            "org/blogtrader/platform/core/netbeans/
             resources/drawingLine.png");
        ImageIcon icon = new ImageIcon(iconImage);
        
        toggleButton = new JToggleButton();
        
        toggleButton.setIcon(icon);
        toggleButton.setToolTipText("Pick Drawing Line");
        
        popup = new JPopupMenu();
        menuItemListener = new MyMenuItemListener();
        
        handledCharts = PersistenceManager.getDefalut()
                        .getAllAvailableHandledChart();
        
        buttonGroup = new ButtonGroup();
        
        for (AbstractHandledChart handledChart : handledCharts) {
            JRadioButtonMenuItem item = 
                new JRadioButtonMenuItem(handledChart.toString());
            item.addActionListener(menuItemListener);
            buttonGroup.add(item);
            popup.add(item);
        }
        
        toggleButton.addItemListener(new ItemListener() {
            public void itemStateChanged(ItemEvent e) {
                int state = e.getStateChange();
                if (state == ItemEvent.SELECTED) {
                    // show popup menu on toggleButton at position:(0, height)
                    popup.show(toggleButton, 0, toggleButton.getHeight());
                }
            }
        });
        
        popup.addPopupMenuListener(new PopupMenuListener() {
            public void popupMenuCanceled(PopupMenuEvent e) {
                toggleButton.setSelected(false);
            }
            public void popupMenuWillBecomeInvisible(PopupMenuEvent e) {
                toggleButton.setSelected(false);
            }
            public void popupMenuWillBecomeVisible(PopupMenuEvent e) {
            }
        });
        
        
        return toggleButton;
    }
    
    private class MyMenuItemListener implements ActionListener {
        public void actionPerformed(ActionEvent ev) {
            JMenuItem item = (JMenuItem)ev.getSource();
            String selectedStr = item.getText();
            
            AnalysisChartTopComponent analysisTc =
                AnalysisChartTopComponent.getSelected();
            
            if (analysisTc == null) {
                return;
            }
            
            AbstractChartViewContainer viewContainer = 
                analysisTc.getSelectedViewContainer();
            AbstractChartView masterView = viewContainer.getMasterView();
            if (!(masterView instanceof WithDrawingPart)) {
                return;
            }
            
            DrawingPart drawingPart = 
                ((WithDrawingPart)masterView).getCurrentDrawing();
            
            if (drawingPart == null) {
                JOptionPane.showMessageDialog(
                        WindowManager.getDefault().getMainWindow(),
                        "Please add a layer firstly to pick line type",
                        "Pick line type",
                        JOptionPane.OK_OPTION,
                        null);
                return;
            }
            
            AbstractHandledChart selectedHandledChart = null;
            
            for (AbstractHandledChart handledChart : handledCharts) {
                if (handledChart.toString().equalsIgnoreCase(selectedStr)) {
                    selectedHandledChart = handledChart;
                    break;
                }
            }
            
            if (selectedHandledChart == null) {
                return;
            }
            
            AbstractHandledChart handledChart = 
                selectedHandledChart.createNewInstance();
            handledChart.setPart(drawingPart);
            drawingPart.setHandledChart(handledChart);
            
            Series masterSeries = viewContainer.getMasterSeries();
            DrawingDescriptor description = 
                viewContainer.getDescriptors().findDrawingDescriptor(
                    drawingPart.getLayerName(),
                    masterSeries.getUnit(),
                    masterSeries.getNUnits());
            
            if (description != null) {
                Node stockNode = analysisTc.getActivatedNodes()[0];
                Node node = 
                    stockNode.getChildren()
                        .findChild(DescriptorGroupNode.DRAWINGS)
                        .getChildren().findChild(description.getDisplayName());
                
                if (node != null) {
                    ViewAction action = 
                        (ViewAction)node.getLookup().lookup(ViewAction.class);
                    assert action != null : 
                        "view action of this layer's node is null!";
                    action.view();
                }
            } else {
                /** best effort, should not happen */
                viewContainer.setCursorCrossVisible(false);
                drawingPart.setActived(true);
                
                SwitchHideShowDrawingLineAction.updateToolbar(viewContainer);
            }
            
        }
    }
    
    
}

How to hide/show toolbar dynamically in NetBeans Platform

To hide/show toolbar dynamically in NetBeans Platform, you should pre-define a toolbar Configuration firstly, then set to what you want via:

ToolbarPool.getDefault().set(String toolbarConfiguratonName);

1. Define toolbar configuration files in module's resource director: blogtrader-platform-branding\src\org\blogtrader\platform\branding\netbeans\resources\Toolbars\

Standard.xml:

<?xml version="1.0"?>
<!DOCTYPE Configuration PUBLIC "-//NetBeans IDE//DTD toolbar//EN"
 "http://www.netbeans.org/dtds/toolbar.dtd">

<Configuration>
    <Row>
        <Toolbar name="View" />
        <Toolbar name="Control" />
        <Toolbar name="Indicator" />
        <Toolbar name="Draw" />
        <Toolbar name="Memory" />
    </Row>
  
    <Row>
        <Toolbar name="File" position="2" visible="false" />
        <Toolbar name="Edit" position="2" visible="false" />
        <Toolbar name="Build" position="2" visible="false" />
        <Toolbar name="Debug" position="2" visible="false" />
        <Toolbar name="Versioning" position="2" visible="false" />
    </Row>
</Configuration>

Developing.xml:

<?xml version="1.0"?>

<!DOCTYPE Configuration PUBLIC "-//NetBeans IDE//DTD toolbar//EN" 
"http://www.netbeans.org/dtds/toolbar.dtd">

<Configuration>
    <Row>
        <Toolbar name="View" />
        <Toolbar name="Control" />
        <Toolbar name="Indicator" />
        <Toolbar name="Draw" />
        <Toolbar name="Memory" />
    </Row>
  
    <Row>
        <Toolbar name="File" position="2" />
        <Toolbar name="Edit" position="2" />
        <Toolbar name="Build" position="2" />
        <Toolbar name="Debug" position="2" visible="false" />
        <Toolbar name="Versioning" position="2" visible="false" />
    </Row>
</Configuration>

2. register the configuration files in layer.xml:<br>

<?xml version="1.0"?>
<!DOCTYPE filesystem PUBLIC "-//NetBeans//DTD Filesystem 1.0//EN"
 "http://www.netbeans.org/dtds/filesystem-1_0.dtd">
<filesystem>    
    <folder name="Toolbars">
    
        <file name="Standard.xml" url="Toolbars/Standard.xml">
        </file>
        <attr name="Standard.xml/Developing.xml" boolvalue="true" />
        <file name="Developing.xml" url="Toolbars/Developing.xml">
        </file>

    </folder >
</filesystem>

3. set to toolbar configuration that you want via:

// change to toolbar layer as Standard
ToolbarPool.getDefault().set("Standard"); 

// change to toolbar layer as Developing
ToolbarPool.getDefault().set("Developing");