For IDE support, an indexer is very important for declaration-find, code-completion, re-factory etc. Indexer will monitor the standard lib and your project, analyzing the source/binary files, gather meta-information of classes, functions, global vars, usages, and store them in a search engine (It's a Lucene back-end in NetBeans). That means, you need to support project management and platform management first, so you know where's the class/source path of platform and your current opened projects.

ErlyBird implemented Erlang platform and project management in modules: erlang.platform and erlang.project, these code are derived from Tor's Ruby module, I've migrated them to CSL, but did not rewrote them in Scala. It's OK for integrating these Java written modules with Scala written erlang.editor module.

First, you should identify the class path type, you can register them in ErlangLanguage.scala as:

    /** @see org.netbeans.modules.erlang.platform.ErlangPlatformClassPathProvider and ModuleInstall */
    override
    def getLibraryPathIds = Collections.singleton(BOOT)

    override
    def getSourcePathIds = Collections.singleton(SOURCE)

object ErlangLanguage {
    val BOOT    = "erlang/classpath/boot"
    val COMPILE = "erlang/classpath/compile"
    val EXECUTE = "erlang/classpath/execute"
    val SOURCE  = "erlang/classpath/source"
}

Where, BOOT, thus getLibraryPathIds will point to Erlang OTP libs' path later in:
org.netbeans.modules.erlang.platform.ErlangPlatformClassPathProvider

Now, the indexer itself, ErlangIndex.scala which extends CSL's EmbeddingIndexer and implemented:

override protected def index(indexable:Indexable, parserResult:Result, context:Context) :Unit

with a factory:

class Factory extends EmbeddingIndexerFactory

Register it in ErlangLanguage.scala as:

    override
    def getIndexerFactory = new ErlangIndexer.Factory

Now, tell the CSL to monitor and index them, for Erlang OTP libs, this is done in ModuleInstall.java:

    @Override
    public void restored() {
        GlobalPathRegistry.getDefault().register(ErlangPlatformClassPathProvider.BOOT, new ClassPath[] { ErlangPlatformClassPathProvider.getBootClassPath() });
    }
    
    @Override
    public void uninstalled() {
        GlobalPathRegistry.getDefault().unregister(ErlangPlatformClassPathProvider.BOOT, new ClassPath[] { ErlangPlatformClassPathProvider.getBootClassPath() });
    }

All classpaths registered in GlobalPathRegistry will be monitored and indexed automatically. As "ModuleInstall.java" is also registered in module erlang.platform's manifest.mf as OpenIDE-Module-Install class, it will be automatically loaded and invoked when this module is installed/activated in NetBeans:

Manifest-Version: 1.0
OpenIDE-Module: org.netbeans.modules.erlang.platform
OpenIDE-Module-Layer: org/netbeans/modules/erlang/platform/resources/layer.xml
OpenIDE-Module-Localizing-Bundle: org/netbeans/modules/erlang/platform/Bundle.properties
OpenIDE-Module-Install: org/netbeans/modules/erlang/platform/ModuleInstall.class
OpenIDE-Module-Specification-Version: 0.18.0

For project's classpath, the "GlobalPathRegistry registering is at RubyProject.java (don't be confused by the name, it's a derivation from Ruby's code, I did not change it yet), as:

    private final class ProjectOpenedHookImpl extends ProjectOpenedHook {
        
        ProjectOpenedHookImpl() {}
        
        protected void projectOpened() {
            // register project's classpaths to GlobalPathRegistry
            ClassPathProviderImpl cpProvider = lookup.lookup(ClassPathProviderImpl.class);
            GlobalPathRegistry.getDefault().register(RubyProject.BOOT, cpProvider.getProjectClassPaths(RubyProject.BOOT));
            GlobalPathRegistry.getDefault().register(RubyProject.SOURCE, cpProvider.getProjectClassPaths(RubyProject.SOURCE));            
        }
        
        protected void projectClosed() {
            // unregister project's classpaths to GlobalPathRegistry
            ClassPathProviderImpl cpProvider = lookup.lookup(ClassPathProviderImpl.class);
            GlobalPathRegistry.getDefault().unregister(RubyProject.SOURCE, cpProvider.getProjectClassPaths(RubyProject.SOURCE));
        }
        
    }

So the project's source path will be registered when this project is opened, thus, trigger the index engine.

Now, when you first install Erlang plugin, IDE will index whole OTP libs once. And when an Erlang project is opened, the project's source files will be indexed too.

Then, you need to do a reverse job: search index, get the meta-data to composite Erlang symbols and AST items. This is done in ErlangIndex.scala, which has helper methods to search the index fields, and fetch back the stored signature string, extract it to symbol, and invoke the parsing manager to resolve the AST items. With the help of ErlangIndex.scala, you can search across the whole libs, find completion items, find declarations, find documents embedded in source comments, find usages.

As the first use of index feature, I've implemented the code-completion and go-to-declaration for global modules functions:

When user input "lists:", will invoke code-completion feature, since it's a remote function call, code-completion will search the index data, and find it's from OTP's "lists.erl", after parsing this file, we get all exported functions AstDfn, so, there are a lot of meta-information can be used now, I'll implement the documents tooltips later.

This will be the final one of the series of Erlang plug-in in Scala blogs, thanks for reading and feedback. I'll take a break during the weekend. From the next Monday, with the spring is coming, I'll join a new project, which is an ambitious financial information platform, I'll bring Scala, Lift, Erlang to it. BTW, congratulations to Lift 1.0!

Implementing Code-Completion is a bit complex, but you can got it work gradually. At the first step, you can implement Code-Completion for local vars/functions only, then, with the indexed supporting, you can add completion for remote functions.

You should define some kinds of completion proposal, which may show different behaviors when they are popped up and guard you followed steps. For example, a function proposal can auto-fill parameters, on the other side, a keyword proposal just complete itself.

The completion proposal classes are defined in  ErlangComplectionProposal.scala, which implemented CSL's interface CompletionProposal. you may notice that the function proposal is the most complex one, which should handle parameters information.

Then, you should implement CSL's interface CodeCompletionHandler, for Erlang, it's  ErlangCodeCompletion, where, the key method is:

    override
    def complete(context:CodeCompletionContext) :CodeCompletionResult = {
        this.caseSensitive = context.isCaseSensitive
        val pResult = context.getParserResult.asInstanceOf[ErlangParserResult]
        val lexOffset = context.getCaretOffset
        val prefix = context.getPrefix match {
            case null => ""
            case x => x
        }

        val kind = if (context.isPrefixMatch) QuerySupport.Kind.PREFIX else QuerySupport.Kind.EXACT
        val queryType = context.getQueryType

        val doc = LexUtil.document(pResult, true) match {
            case None => return CodeCompletionResult.NONE
            case Some(x) => x.asInstanceOf[BaseDocument]
        }

        val proposals = new ArrayList[CompletionProposal]
        val completionResult = new DefaultCompletionResult(proposals, false)

        // Read-lock due to Token hierarchy use
        doc.readLock
        try {
            val astOffset = LexUtil.astOffset(pResult, lexOffset)
            if (astOffset == -1) {
                return CodeCompletionResult.NONE
            }
            val root = pResult.rootScope match {
                case None => return CodeCompletionResult.NONE
                case Some(x) => x
            }
            val th = LexUtil.tokenHierarchy(pResult).get
            val fileObject = LexUtil.fileObject(pResult).get

            val request = new CompletionRequest
            request.completionResult = completionResult
            request.result = pResult
            request.lexOffset = lexOffset
            request.astOffset = astOffset
            request.index = ErlangIndex.get(pResult)
            request.doc = doc
            request.info = pResult
            request.prefix = prefix
            request.th = th
            request.kind = kind
            request.queryType = queryType
            request.fileObject = fileObject
            request.anchor = lexOffset - prefix.length
            request.root = root
            ErlangCodeCompletion.request = request
            
            val token = LexUtil.token(doc, lexOffset - 1) match {
                case None => return completionResult
                case Some(x) => x
            }

            token.id match {
                case ErlangTokenId.LineComment =>
                    // TODO - Complete symbols in comments?
                    return completionResult
                case ErlangTokenId.StringLiteral =>
                    //completeStrings(proposals, request)
                    return completionResult
                case _ =>
            }
            
            val ts = LexUtil.tokenSequence(th, lexOffset - 1) match {
                case None => return completionResult
                case Some(x) =>
                    x.move(lexOffset - 1)
                    if (!x.moveNext && !x.movePrevious) {
                        return completionResult
                    }
                    x
            }
 
            val closetToken = LexUtil.findPreviousNonWsNonComment(ts)

            if (root != null) {
                val sanitizedRange = pResult.sanitizedRange
                val offset = if (sanitizedRange != OffsetRange.NONE && sanitizedRange.containsInclusive(astOffset)) {
                    sanitizedRange.getStart
                } else astOffset

                val call = Call(null, null, false)
                findCall(root, ts, th, call, 0)
                val prefixBak = request.prefix
                call match {
                    case Call(null, _, _) =>
                    case Call(base, _, false) =>
                        // it's not a call, but may be candicate for module name, try to get modules and go-on
                        completeModules(base, proposals, request)
                    case Call(base, select, true) =>
                        if (select != null) {
                            request.prefix = call.select.text.toString
                        } else {
                            request.prefix = ""
                        }
                        completeModuleFunctions(call.base, proposals, request)
                        // Since is after a ":", we won't added other proposals, just return now whatever
                        return completionResult
                }
                request.prefix = prefixBak
                completeLocals(proposals, request)
            }

            completeKeywords(proposals, request)
        } finally {
            doc.readUnlock
        }

        completionResult
    }

For a Erlang function call, you should check the tokens surrounding the caret to get the call's base name and select first, which is done by a method findCall:

    private def findCall(rootScope:AstRootScope, ts:TokenSequence[TokenId], th:TokenHierarchy[_], call:Call, times:Int) :Unit = {
        assert(rootScope != null)
        val closest = LexUtil.findPreviousNonWsNonComment(ts)
        val idToken = closest.id match {
            case ErlangTokenId.Colon =>
                call.caretAfterColon = true
                // skip RParen if it's the previous
                if (ts.movePrevious) {
                    val prev = LexUtil.findPreviousNonWs(ts)
                    if (prev != null) {
                        prev.id match {
                            case ErlangTokenId.RParen   => LexUtil.skipPair(ts, ErlangTokenId.LParen,   ErlangTokenId.RParen,   true)
                            case ErlangTokenId.RBrace   => LexUtil.skipPair(ts, ErlangTokenId.LBrace,   ErlangTokenId.RBrace,   true)
                            case ErlangTokenId.RBracket => LexUtil.skipPair(ts, ErlangTokenId.LBracket, ErlangTokenId.RBracket, true)
                            case _ =>
                        }
                    }
                }
                LexUtil.findPrevIncluding(ts, LexUtil.CALL_IDs)
            case id if LexUtil.CALL_IDs.contains(id) => closest
            case _ => null
        }

        if (idToken != null) {
            times match {
                case 0 if call.caretAfterColon => call.base = idToken
                case 0 if ts.movePrevious => LexUtil.findPreviousNonWsNonComment(ts) match {
                        case null => call.base = idToken
                        case prev if prev.id == ErlangTokenId.Colon =>
                            call.caretAfterColon = true
                            call.select = idToken
                            findCall(rootScope, ts, th, call, times + 1)
                        case _ => call.base = idToken
                    }
                case _ => call.base = idToken
            }
        }
    }

    case class Call(var base:Token[TokenId], var select:Token[TokenId], var caretAfterColon:Boolean)

To complete a remote function call, you may need to visit outer modules, which needs an indexer, so as the first step, you can just ignore it, go straight to complete local vars/functions, or keywords:

    private def completeLocals(proposals:List[CompletionProposal], request:CompletionRequest) :Unit = {
        val prefix = request.prefix
        val kind = request.kind
        val pResult = request.result

        val root = request.root
        val closestScope = root.closestScope(request.th, request.astOffset) match {
            case None => return
            case Some(x) => x
        }
        val localVars = closestScope.visibleDfns(ElementKind.VARIABLE)
        localVars ++= closestScope.visibleDfns(ElementKind.PARAMETER)
        localVars.filter{v => filterKind(kind, prefix, v.name)}.foreach{v =>
            proposals.add(new PlainProposal(v, request.anchor))
        }

        val localFuns = closestScope.visibleDfns(ElementKind.METHOD)
        localFuns.filter{f => filterKind(kind, prefix, f.name)}.foreach{f =>
            proposals.add(new FunctionProposal(f, request.anchor))
        }
    }

    private def completeKeywords(proposals:List[CompletionProposal], request:CompletionRequest) :Unit = {
        val prefix = request.prefix
        val itr = LexerErlang.ERLANG_KEYWORDS.iterator
        while (itr.hasNext) {
            val keyword = itr.next
            if (startsWith(keyword, prefix)) {
                proposals.add(new KeywordProposal(keyword, null, request.anchor))
            }
        }
    }

There is a function def visibleDfns(kind: ElementKind): ArrayBuffer[AstDfn] in AstRootScope.scala, if you've put definition items properly in scopes, it should handle the visibility automatically.

Now, register it in  ErlangLanguage.scala:

    override
    def getCompletionHandler = new ErlangCodeCompletion

As usual, run it, you got:

The local functions and vars are proposed plus the keywords. BTW, I've fixed this feature for Scala plug-in.

Case class in Scala is a neat feature, it automatically generates apply, extract and equals functions. But I encountered a strange "equals" behavior today, which wasted me 2 hours to find why.

Define a simple case class "Symbol", then a "VarSymbol" which extends "Symbol" with one more field "name:String". We know you can compare the equality of instances of "VarSymbol" by this "name", Scala automatically gets it right. And of course, a direct instance from "Symbol()" should not equal to any "VarSymbol", since it lacks "name" field, that's right.

case class Symbol()
case class VarSymbol(name:String) extends Symbol

Then, I need a singleton object "NoSymbol", I defined it as:

object NoSymbol extends Symbol

I of course thought this "NoSymbol" should not equal any "VarSymbol" instance, or,

NoSymbol == VarSymbol("I'm var")

Should return false. But life is not so straightforward, it returns true in real life !!!

I finally got my code working, by adding a "case" before "object NoSymbol extends Symbol". This "NoSymbol" won't equal any "VarSymbol" now, I'm grad things finally go back in track.

I don't know if it's a Scala bug, or, that's what Scala thinks it should be: all "Symbol" and it's inherited instances should equal "NoSymbol" object. Anyway, here's the whole code for test:

case class Symbol()
case class VarSymbol(name:String) extends Symbol

object NoSymbol extends Symbol
case object CasedNoSymbol extends Symbol

object TestCaseObject {
    
    def test = {
        val noSym = NoSymbol
        val caseNoSym = CasedNoSymbol
        val varSym = VarSymbol("I'm var")

        if (noSym == varSym) println("NoSym equals varSym !") 
        else println("NoSym doesn't equal varSym")
        
        if (caseNoSym == varSym) println("CaseNoSym equals varSym !") 
        else println("CaseNoSym doesn't equal varSym")
    }
    
}

Run TestCaseObject.test, I got:

NoSym equals varSym !
CaseNoSym doesn't equal varSym

NetBeans 6.7M2 is going to be public available soon, the new look&feel on Mac OS is very awesome. BTW, -Dapple.awt.graphics.UseQuartz=true is a must option for best font-rendering on my macbook. I added it to netbeans.conf as -J-Dapple.awt.graphics.UseQuartz=true. Here is a snapshot of my current screen when working on Erlang plugin in Scala:

Click on the picture to enlarge it:

There is a Java class which takes type parameter T, T is any type:

package dcaoyuan.test;

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;

public class AList<T> implements Iterator<T> {

   List<T> a = new ArrayList<T>();

   {
       a.add((T) "string");
   }

   Iterator<T> itr = a.iterator();

   public boolean hasNext() {
       return itr.hasNext();
   }

   public T next() {
       return itr.next();
   }

   public void remove() {
       throw new UnsupportedOperationException("Not supported yet.");
   }
}

Which implemented java.util.Iterator with type parametered T next(). Notice here next() will return T

The above code simulates a case, that an existed Java lib may return an instance that has type parameter erased, on run-time.

Now let's try to call it in Scala:

package dcaoyuan.test

object Main {

   def main(args: Array[String]) :Unit = {
       testAList_work
       testAList_unwork
   }

   def testAList_work :Unit = {
       val a :AList[_] = new AList
       // a is inferred as AList[Any]
       while (a.hasNext) {
           val e = a.next
           println(e)
       }
   }

   def testAList_unwork :Unit = {
       val a = new AList
       // a is inferred as AList[Nothing]
       while (a.hasNext) {
           // now a.next is Nothing, can not cast to "val e", which should at least be Any
           val e = a.next // will throw java.lang.ClassCastException
           println(e)
       }
   }
}

I wrote two functions: "testAList_work" and "testAList_unwork"

The issue is in testAList_unwork, as a simple "val a = new AList", "a" will be inferred by scalac as AList[Nothing], so, "a.next" will return an instance of type Nothing, then, "val e = a.next" will throw java.lang.ClassCastException

My question is, should this corner case be checked by scalac as an error or warning? So I do not need to dig the clause when it happens on run-time.

The progress of rewriting Erlang plugin for NetBeans in Scala has reached a phase, that the Editor itself works smooth and better than ErlyBird now, the next step is to integrate an Erlang project management and index the modules/functions of OTP/project to provide smart auto-completion.

Now Scala has been proved that it can be integrated into an existed large Java based framework (NetBeans IDE here) without no much problems, I'll begin to rewrite Scala plugin in Scala soon.

Although in most cases, Scala can call existed Java code or classes smoothly, there are still some corner cases. I'll record these corner cases in blogs. Here's the first one which I also posted on scala-user mailing-list, but have not yet got final answer.

Let's begin with a detailed example:

There is a Java interface A:

public interface A<T extends String> {
   void run(T t);
}

Which has a type parameter <T extends String> and abstract method run(T t)

Then a Java abstract class B extended A. But, B, as it, omitted type parameter from A. This is unsafe but valid in Java:

public abstract class B implements A {
   public String me() {
       return "I'm B";
   }
}

Assume above classes A and B have been compiled under javac, and packed in a jar library, and I can not patch it anymore. Now I need to write a class S in Scala which should extend B:

class S extends B {
   override
   def run[T <: String](t:T) = {println(t)}
}

scalac will complain as:

/Users/dcaoyuan/NetBeansProjects/ScalaTestCase/src/S.scala:1: error:
class S needs to be abstract, since method run in trait A of type
(T)Unit is not defined
class S extends B {
/Users/dcaoyuan/NetBeansProjects/ScalaTestCase/src/S.scala:3: error:
method run overrides nothing
   def run[T <: String](t:T) = {println(t)}

I than tried "forSome" type:

class S extends B {
   override
   def run(t:T forSome {type T <: String}) = {println(t)}
}

The code still did not work.

It seems that, since B omitted A's type parameter T, I have no way to get what is T, and can not successfully implement "run(T t)" method.

I also tried other forms of "forSome" usages, and always failed.

But I think Scala can always be saved with mixed Java/Scala code in such corner case, that's what I believed. So, I thought about it later when my brain was spare, and finally got a solution:

I wrote another Java abstract class B1 which extends B and pretended to have implemented "run(T t)", but actually called another new abstract method "runImpl(String t)"

public abstract class B1 extends B {

    public void run(String t) {
        runImpl(t);
    }

    public abstract void runImpl(String t);
}

Now I can let Scala S extends B1 and implement "runImpl(String t)" instead of extending B and implementing "run(T t)".

class S extends B1 {
    override
    def runImpl(t:String) = {println(t)}
}

Yes, scalac won't complain about "runImpl(t:String)" at all, and I got S successfully extends B by bridge class B1.

But I still hope scalac can resolve it directly, with a warning message instead of failing to compile it.

It seems I'm the only one who is committing to hg.netbeans.org/main/contrib these days. Anyway.

Implementing Instant Rename and Go to Declaration is not difficult with AstDfn/AstRef/AstScope. Here is the code:

• Instant Rename is implemented by ErlangInstantRenamer.scala
• Go to Declaration is implemented by ErlangDeclarationFinder.scala

And register them in :

    override
    def getInstantRenamer = new ErlangInstantRenamer

    override
    def getDeclarationFinder = new ErlangDeclarationFinder

Only local function definitions can be traveled to at this time. To jump to remote function definitions, I have to implement an indexer first.

Now let's go on the complex part: Pretty Formatting and Pair Matching. I say they are complex, not because these features are much heavier on language's semantic complex. Implementing these features mostly deals with lexer tokens. But it's a bit brain-dried work to across forward/backward in the token stream to get the pair matching and pretty formatting working as you expected.

Because of the complex, I won't describe the details of how to implement them for Erlang, I just put the links to these source code:

• Pretty Formatting is implemented by ErlangFormatter.scala
• Pair Matching is implemented by ErlangKeystrokeHandler.scala

And registered them in ErlangLanguage.scala as:

    override
    def getKeystrokeHandler = new ErlangKeystrokeHandler

    override
    def hasFormatter =  true

    override
    def getFormatter = new ErlangFormatter

With these feature implemented, the new plugin can automatically complete/match braces and pair, indent properly when you hit BREAK, input a "end" etc.

BTW, the navigator window was improved during these days, it can now properly show the arity/args of each functions. It's done by improved AstNodeVisitor.scala and AstDfn.scala

During the AST node visiting (AstNodeVisitor.scala), I've gathered a lot of variable definitions and their references, we can now try to get editor to mark these occurrences. There are preliminary functions in AstScope.scala, such as findOccurrences(AstItem). You can override AstDfn#isReferredBy(AstRef) and AstRef#isOccurence(AstRef) to get accurate reference relations. As the first step, I just simply judge the reference relation by the equation of names.

We'll extends org.netbeans.modules.csl.api.OccurrencesFinder and implement:

run(pResult:ErlangParserResult, event:SchedulerEvent) :Unit

First, we try to get the AstItem which is at the caretPosition by rootScope.findItemAt(th, caretPosition), and verify if it's a valid token.

Then, by calling rootScope.findOccurrences(item), we collect all occurrences of this item, and put a ColoringAttributes.MARK_OCCURRENCES with its OffsetRange.

The code of ErlangOccurrencesFinder.scala:

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 *
 * Contributor(s):
 *
 * Portions Copyrighted 2009 Sun Microsystems, Inc.
 */
package org.netbeans.modules.erlang.editor

import _root_.java.util.{HashMap,List,Map}
import javax.swing.text.Document
import org.netbeans.api.lexer.{Token,TokenId,TokenHierarchy}
import _root_.org.netbeans.modules.csl.api.{ColoringAttributes,OccurrencesFinder,OffsetRange}
import org.netbeans.modules.parsing.spi.Parser
import org.netbeans.modules.parsing.spi.{Scheduler,SchedulerEvent}
import org.netbeans.modules.erlang.editor.ast.{AstDfn,AstItem,AstRef,AstRootScope}
import org.netbeans.modules.erlang.editor.lexer.{ErlangTokenId,LexUtil}
import org.openide.filesystems.FileObject

/**
 *
 * @author Caoyuan Deng
 */
class ErlangOccurrencesFinder extends OccurrencesFinder[ErlangParserResult] {

    protected var cancelled = false
    private var caretPosition = 0
    private var occurrences :Map[OffsetRange, ColoringAttributes] = _
    private var file :FileObject = _

    protected def isCancelled = synchronized {cancelled}

    protected def resume :Unit = synchronized {cancelled = false}

    override
    def getPriority = 0

    override
    def getSchedulerClass = Scheduler.CURSOR_SENSITIVE_TASK_SCHEDULER

    override
    def getOccurrences :Map[OffsetRange, ColoringAttributes] = occurrences

    override
    def cancel :Unit = synchronized {cancelled = true}

    override
    def setCaretPosition(position:Int) :Unit = {this.caretPosition = position}

    def run(pResult:ErlangParserResult, event:SchedulerEvent) :Unit = {
        resume
        if (pResult == null || isCancelled) {
            return
        }

        val currentFile = pResult.getSnapshot.getSource.getFileObject
        if (currentFile != file) {
            // Ensure that we don't reuse results from a different file
            occurrences = null
            file = currentFile
        }

        for (rootScope <- pResult.rootScope;
             th <- LexUtil.tokenHierarchy(pResult);
             doc <- LexUtil.document(pResult, true);
             // * we'll find item by offset of item's idToken, so, use caretPosition directly
             item <- rootScope.findItemAt(th, caretPosition);
             idToken <- item.idToken
        ) {
            var highlights = new HashMap[OffsetRange, ColoringAttributes](100)

            val astOffset = LexUtil.astOffset(pResult, caretPosition)
            if (astOffset == -1) {
                return
            }

            // * When we sanitize the line around the caret, occurrences
            // * highlighting can get really ugly
            val blankRange = pResult.sanitizedRange

            if (blankRange.containsInclusive(astOffset)) {
                return
            }

            // * test if document was just closed?
            LexUtil.document(pResult, true) match {
                case None => return
                case _ =>
            }

            try {
                doc.readLock
                val length = doc.getLength
                val astRange = LexUtil.rangeOfToken(th.asInstanceOf[TokenHierarchy[TokenId]], idToken.asInstanceOf[Token[TokenId]])
                val lexRange = LexUtil.lexerOffsets(pResult, astRange)
                var lexStartPos = lexRange.getStart
                var lexEndPos = lexRange.getEnd

                // If the buffer was just modified where a lot of text was deleted,
                // the parse tree positions could be pointing outside the valid range
                if (lexStartPos > length) {
                    lexStartPos = length
                }
                if (lexEndPos > length) {
                    lexEndPos = length
                }

                LexUtil.token(doc, caretPosition) match {
                    case None => return
                    case token => // valid token, go on
                }
            } finally {
                doc.readUnlock
            }

            val _occurrences = rootScope.findOccurrences(item)
            for (_item <- _occurrences;
                 _idToken <- _item.idToken
            ) {
                highlights.put(LexUtil.rangeOfToken(th.asInstanceOf[TokenHierarchy[TokenId]],
                                                    _idToken.asInstanceOf[Token[TokenId]]),
                               ColoringAttributes.MARK_OCCURRENCES)
            }

            if (isCancelled) {
                return
            }

            if (highlights.size > 0) {
                val translated = new HashMap[OffsetRange, ColoringAttributes](2 * highlights.size)
                val entries = highlights.entrySet.iterator
                while (entries.hasNext) {
                    val entry = entries.next
                    LexUtil.lexerOffsets(pResult, entry.getKey) match {
                        case OffsetRange.NONE =>
                        case range => translated.put(range, entry.getValue)
                    }
                }

                highlights = translated

                this.occurrences = highlights
            } else {
                this.occurrences = null
            }
        }
    }
}

Finally, again, register it in ErlangLanguage.scala

override def hasOccurrencesFinder = true
override def getOccurrencesFinder = new ErlangOccurrencesFinder

And add mark-occurrences effect setting in fontsColors.xml:

<fontcolor name="mark-occurrences" bgColor="ECEBA3"/>

That's all. Run it, you got:

Var LogPid's definition and references were highlighted now.

The correctness of occurrences marking depends on the correctness of scopes of variables when you visit AST node, I did not finish all of such work yet, but, enabling occurrences marking feature is very helpful for further work.

With more detailed AstNodeVisitor.scala, I got the semantic information of function calls, variable definitions and references etc. It's time to implement CSL's SemanticAnalyzer, which is the entrance of semantic highlighting.

I then encountered a Scala's corner case issue :-(

SemanticAnalyzer.java is a sub-class of ParserResultTask:

package org.netbeans.modules.parsing.spi;
public abstract class ParserResultTask<T extends Parser.Result> extends SchedulerTask {
    public abstract void run (T result, SchedulerEvent event);
    public abstract int getPriority ();
}

ParserResultTask's signature in class file is:

<<TLorg/netbeans/modules/parsing/spi/Parser$Result;>Lorg/netbeans/modules/parsing/spi/SchedulerTask;>

run's method signature:

<(TT;Lorg/netbeans/modules/parsing/spi/SchedulerEvent;)V>

ErlangSemanticAnalyzer extended SemanticAnalyzer, so I should implement:

void run (T result, SchedulerEvent event);

which carries type parameter T from ParserResultTask<T extends Parser.Result>. But unfortunately, SemanticAnalyzer extends ParserResultTask as:

public abstract class SemanticAnalyzer extends ParserResultTask;

That is, SemanticAnalyzer erased type parameter of its parent. It's valid in Java. But in Scala, I can not successfully extend SemanticAnalyzer anymore, since when I tried to implement "void run (T result, SchedulerEvent event)", the scalac always complained that I did not override this method with type T. It seems scalac checks type parameter deeply, and needs sub-class to always explicitly extend parent class with type parameter. Since SemanticAnalyzer has erased type parameter, even I tried to implement "run" as:

override def run[T <: Parser.Result](result:T, event:SchedulerEvent) :Unit 

scalac still complained about it.

I have no idea of how to bypass this issue. I then patched SemanticAnalyzer.java, let it carries same type parameter as its parent:

public abstract class SemanticAnalyzer<T extends Parser.Result> extends ParserResultTask<T> 

Now everything works. My final code is:

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2008 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 *
 * Contributor(s):
 *
 * Portions Copyrighted 2009 Sun Microsystems, Inc.
 */
package org.netbeans.modules.erlang.editor

import _root_.java.util.{HashMap,HashSet,Map,Set}
import javax.swing.text.Document
import org.netbeans.api.lexer.Token
import org.netbeans.api.lexer.TokenHierarchy
import org.netbeans.api.lexer.TokenId
import org.netbeans.modules.csl.api.{ColoringAttributes,ElementKind,OffsetRange,SemanticAnalyzer}
import org.netbeans.modules.parsing.spi.Parser
import org.netbeans.modules.parsing.spi.{Scheduler,SchedulerEvent,ParserResultTask}
import org.netbeans.modules.erlang.editor.ast.{AstDfn,AstItem,AstRef,AstRootScope}
import org.netbeans.modules.erlang.editor.lexer.LexUtil
import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId

/**
 *
 * @author Caoyuan Deng
 */
class ErlangSemanticAnalyzer extends SemanticAnalyzer[ErlangParserResult] {

    private var cancelled = false
    private var semanticHighlights :Map[OffsetRange, Set[ColoringAttributes]] = _

    protected def isCancelled :Boolean = synchronized {cancelled}

    protected def resume :Unit = synchronized {cancelled = false}

    override
    def getHighlights :Map[OffsetRange, Set[ColoringAttributes]] = semanticHighlights

    override
    def getPriority = 0

    override
    def getSchedulerClass = Scheduler.EDITOR_SENSITIVE_TASK_SCHEDULER
    
    override
    def cancel :Unit = {cancelled = true}

    @throws(classOf[Exception])
    override
    def run(pResult:ErlangParserResult, event:SchedulerEvent) :Unit = {
        resume
        semanticHighlights = null

        if (pResult == null || isCancelled) {
            return
        }

        for (rootScope <- pResult.rootScope;
             th <- LexUtil.tokenHierarchy(pResult);
             doc <- LexUtil.document(pResult, true)
        ) {
            var highlights = new HashMap[OffsetRange, Set[ColoringAttributes]](100)
            visitItems(th.asInstanceOf[TokenHierarchy[TokenId]], rootScope, highlights)

            this.semanticHighlights = if (highlights.size > 0) highlights else null
        }
    }

    private def visitItems(th:TokenHierarchy[TokenId], rootScope:AstRootScope, highlights:Map[OffsetRange, Set[ColoringAttributes]]) :Unit = {
        import ElementKind._
        for (item <- rootScope.idTokenToItem(th).values;
             hiToken <- item.idToken
        ) {
            
            val hiRange = LexUtil.rangeOfToken(th, hiToken.asInstanceOf[Token[TokenId]])
            item match {
                case dfn:AstDfn => dfn.getKind match {
                        case MODULE =>
                            highlights.put(hiRange, ColoringAttributes.CLASS_SET)
                        case CLASS =>
                            highlights.put(hiRange, ColoringAttributes.CLASS_SET)
                        case ATTRIBUTE =>
                            highlights.put(hiRange, ColoringAttributes.STATIC_SET)
                        case METHOD =>
                            highlights.put(hiRange, ColoringAttributes.METHOD_SET)
                        case PARAMETER =>
                            highlights.put(hiRange, ColoringAttributes.PARAMETER_SET)
                        case _ =>
                    }
                case ref:AstRef => ref.getKind match {
                        case METHOD =>
                            highlights.put(hiRange, ColoringAttributes.FIELD_SET)
                        case PARAMETER =>
                            highlights.put(hiRange, ColoringAttributes.PARAMETER_SET)
                        case _ =>
                    }
            }
        }
    }
}

In ErlangSemanticeAnalyzer, what you need to do is traversing all AstDfn and AstRef instances, and give them a proper ColoringAttributes set. I marked all functions as METHOD_SET(mod-method), attributes as STATIC_SET(mod-static), and function call names as FIELD_SET(mod-field), parameters as PARAMETER_SET(mod-parameter), the names in parenthesis are the corresponding names that in fontColors.xml setting file, you can define the highlighting effects in this xml file.

Again, don't forget to register it in ErlangLanguage.scala:

override def getSemanticAnalyzer = new ErlangSemanticAnalyzer

Run it, and I got highlighted source code as:

The function declaration names are underline, parameters are bold, function calls are bold too.

I'll gather even more detailed semantic information later, so I can identity the unused variable definitions and var references without definitions etc.

During the weekend, I've done some preliminary error recover work for Erlang's rats! definition. Now I'll go on some features based on analysis on AST tree. As the simplest step, we'll visit/analyze AST tree to get the structure information, use them for Navigator window and code folding.

First, we need to record the Structure information in some way. Each language has different AST tree and may be generated by varies tools, the AST element should be wrapped in some more generic classes to get integrated into CSL framework. There is an interface "org.netbeans.modules.csl.api.ElementKind", which is used for this purpose. But it's not enough, we need some facilities to not only wrap AST element, and also help us to identify the element as a definition or reference, in which visible scope etc.

I wrote a couple of these facilities when I wrote Erlang/Fortress/Scala plug-ins, these facilities can be repeatedly used for other languages by minor modifying. They are AstItem, AstDfn, AstRef, AstScope, AstRootScope. For Erlang plugin, you can find them under director: erlang.editor/src/org/netbeans/modules/erlang/editor/ast

AstDfn.scala is used to store a definition, such as definitions of class, module, method, variable, attribute etc. AstRef.scala is used to store reference that refers to definition, for example, the usages of a class, a variable, a method call etc.

There should be some lexer level utilities too, to help you get the corresponding tokens when you visit AST tree. It's LexUtil.scala, which will be also heavy used for code-folding, indentation ... features.

With above facilities, we can visit an AST tree now, I'll do the simplest task first: get all functions/attribues name and bounds tokens.

AstNodeVisitor.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 * 
 * Copyright 2008 Sun Microsystems, Inc. All rights reserved.
 * 
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 * 
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 * 
 * Contributor(s):
 * 
 * Portions Copyrighted 2009 Sun Microsystems, Inc.
 */
package org.netbeans.modules.erlang.editor.node

import org.netbeans.api.lexer.{Token, TokenId, TokenHierarchy, TokenSequence}
import org.netbeans.modules.csl.api.ElementKind

import org.netbeans.modules.erlang.editor.ast.{AstDfn, AstItem, AstRef, AstRootScope, AstScope, AstVisitor}
import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId._
import org.netbeans.modules.erlang.editor.lexer.{ErlangTokenId, LexUtil}
import org.openide.filesystems.FileObject

import scala.collection.mutable.ArrayBuffer

import xtc.tree.GNode
import xtc.tree.Node
import xtc.util.Pair

/**
 *
 * @author Caoyuan Deng
 */
class AstNodeVisitor(rootNode:Node, th:TokenHierarchy[_], fo:FileObject) extends AstVisitor(rootNode, th) {

    def visitS(that:GNode) = {
        val formNodes = that.getList(0).iterator
        while(formNodes.hasNext) {
            visitForm(formNodes.next)
        }
    }

    def visitForm(that:GNode) = {
        enter(that)

        val scope = new AstScope(boundsTokens(that))
        rootScope.addScope(scope)

        scopes.push(scope)
        visitNodeOnly(that.getGeneric(0))
        
        exit(that)
        scopes.pop
    }


    def visitAttribute(that:GNode) = {
        that.get(0) match {
            case atomId:GNode =>
                val attr = new AstDfn(that, idToken(idNode(atomId)), scopes.top, ElementKind.ATTRIBUTE, fo)
                rootScope.addDfn(attr)
        }
    }

    def visitFunction(that:GNode) = {
        visitFunctionClauses(that.getGeneric(0))
    }

    def visitFunctionClauses(that:GNode) = {
        val fstClauseNode = that.getGeneric(0)
        visitFunctionClause(fstClauseNode)
    }

    def visitFunctionClause(that:GNode) = {
        val id = idNode(that.getGeneric(0))
        val fun = new AstDfn(that, idToken(id), scopes.top, ElementKind.METHOD, fo)
        rootScope.addDfn(fun)
    }

    def visitRule(that:GNode) = {

    }
}

As you can see, I just visit function/attribute related AST nodes, and gather all function/attribute declarations (or, definitions). My AST tree is generated by rats!, so the code looks like above. If you are using other parsers, I assume you know of course how to travel it.

AstNodeVisitor extends AstVisitor, I wrote some helper methods in AstVisitor.scala, for example, getting the bounds tokens for a definition/scope.

Now, you need to put the AST visiting task to ErlangParser.scala, so, it will be called when parsing finished, and you'll get an AST tree. The code is something like:

    private def analyze(context:Context) :Unit = {
        val doc = LexUtil.document(context.snapshot, false)

        // * we need TokenHierarchy to do anaylzing task
        for (root <- context.root;
             th <- LexUtil.tokenHierarchy(context.snapshot)) {
            // * Due to Token hierarchy will be used in analyzing, should do it in an Read-lock atomic task
            for (x <- doc) {x.readLock}
            try {
                val visitor = new AstNodeVisitor(root, th, context.fo)
                visitor.visit(root)
                context.rootScope = Some(visitor.rootScope)
            } catch {
                case ex:Throwable => ex.printStackTrace
            } finally {
                for (x <- doc) {x.readUnlock}
            }
        }
    }

The rootScope will be carried in ErlangParserResult.scala.

With the visitor's rootScope, we can implement the navigator window and code folding now. What you need is to implement an org.netbeans.modules.csl.api.StructureScanner. My implementation is: ErlangStructureAnalyzer.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 *
 * Contributor(s):
 *
 * Portions Copyrighted 2009 Sun Microsystems, Inc.
 */
package org.netbeans.modules.erlang.editor;

import _root_.java.util.{ArrayList,Collections,HashMap,List,Map,Set,Stack}
import javax.swing.ImageIcon;
import javax.swing.text.{BadLocationException,Document}
import org.netbeans.api.lexer.{Token,TokenId,TokenHierarchy,TokenSequence}
import org.netbeans.editor.{BaseDocument,Utilities}
import org.netbeans.modules.csl.api.{ElementHandle,ElementKind,HtmlFormatter,Modifier,OffsetRange,StructureItem,StructureScanner}
import org.netbeans.modules.csl.api.StructureScanner._
import org.netbeans.modules.csl.spi.ParserResult
import org.netbeans.modules.erlang.editor.ast.{AstDfn,AstRootScope,AstScope}
import org.netbeans.modules.erlang.editor.lexer.{ErlangTokenId,LexUtil}
import org.openide.util.Exceptions

import scala.collection.mutable.ArrayBuffer

/**
 *
 * @author Caoyuan Deng
 */
class ErlangStructureAnalyzer extends StructureScanner {

    override
    def getConfiguration :Configuration = null

    override
    def scan(result:ParserResult) :List[StructureItem] = result match {
        case null => Collections.emptyList[StructureItem]
        case pResult:ErlangParserResult => 
            var items = Collections.emptyList[StructureItem]
            for (rootScope <- pResult.rootScope) {
                items = new ArrayList[StructureItem](rootScope.dfns.size)
                scanTopForms(rootScope, items, pResult)
            }
            items
    }

    private def scanTopForms(scope:AstScope, items:List[StructureItem], pResult:ErlangParserResult) :Unit = {
        for (dfn <- scope.dfns) {
            dfn.getKind match {
                case ElementKind.ATTRIBUTE | ElementKind.METHOD => items.add(new ErlangStructureItem(dfn, pResult))
                case _ =>
            }
            scanTopForms(dfn.bindingScope, items, pResult)
        }
    }

    override
    def folds(result:ParserResult) :Map[String, List[OffsetRange]] = result match {
        case null => Collections.emptyMap[String, List[OffsetRange]]
        case pResult:ErlangParserResult =>
            var folds = Collections.emptyMap[String, List[OffsetRange]]
            for (rootScope <- pResult.rootScope;
                 doc <- LexUtil.document(pResult, true);
                 th <- LexUtil.tokenHierarchy(pResult);
                 ts <- LexUtil.tokenSequence(th, 1)
            ) {
                folds = new HashMap[String, List[OffsetRange]]
                val codefolds = new ArrayList[OffsetRange]
                folds.put("codeblocks", codefolds); // NOI18N

                // * Read-lock due to Token hierarchy use
                doc.readLock

                addCodeFolds(pResult, doc, rootScope.dfns, codefolds)

                var lineCommentStart = 0
                var lineCommentEnd = 0
                var startLineCommentSet = false

                val comments = new Stack[Array[Integer]]
                val blocks = new Stack[Integer]

                while (ts.isValid && ts.moveNext) {
                    val token = ts.token
                    token.id match {
                        case ErlangTokenId.LineComment =>
                            val offset = ts.offset
                            if (!startLineCommentSet) {
                                lineCommentStart = offset
                                startLineCommentSet = true
                            }
                            lineCommentEnd = offset

                        case ErlangTokenId.Case | ErlangTokenId.If | ErlangTokenId.Try | ErlangTokenId.Receive =>
                            val blockStart = ts.offset
                            blocks.push(blockStart)

                            startLineCommentSet = false
                        
                        case ErlangTokenId.End if !blocks.empty =>
                            val blockStart = blocks.pop.asInstanceOf[Int]
                            val blockRange = new OffsetRange(blockStart, ts.offset + token.length)
                            codefolds.add(blockRange)

                            startLineCommentSet = false
                        case _ =>
                            startLineCommentSet = false
                    }
                }

                doc.readUnlock

                try {
                    /** @see GsfFoldManager#addTree() for suitable fold names. */
                    lineCommentEnd = Utilities.getRowEnd(doc, lineCommentEnd)

                    if (Utilities.getRowCount(doc, lineCommentStart, lineCommentEnd) > 1) {
                        val lineCommentsFolds = new ArrayList[OffsetRange];
                        val range = new OffsetRange(lineCommentStart, lineCommentEnd)
                        lineCommentsFolds.add(range)
                        folds.put("comments", lineCommentsFolds) // NOI18N
                    }
                } catch {
                    case ex:BadLocationException => Exceptions.printStackTrace(ex)
                }
            }

            folds
    }

    @throws(classOf[BadLocationException])
    private def addCodeFolds(pResult:ErlangParserResult, doc:BaseDocument, defs:ArrayBuffer[AstDfn], codeblocks:List[OffsetRange]) :Unit = {
        import ElementKind._
       
        for (dfn <- defs) {
            val kind = dfn.getKind
            kind match {
                case FIELD | METHOD | CONSTRUCTOR | CLASS | MODULE | ATTRIBUTE =>
                    var range = dfn.getOffsetRange(pResult)
                    var start = range.getStart
                    // * start the fold at the end of the line behind last non-whitespace, should add 1 to start after "->"
                    start = Utilities.getRowLastNonWhite(doc, start) + 1
                    val end = range.getEnd
                    if (start != -1 && end != -1 && start < end && end <= doc.getLength) {
                        range = new OffsetRange(start, end)
                        codeblocks.add(range)
                    }
                case _ =>
            }
    
            val children = dfn.bindingScope.dfns
            addCodeFolds(pResult, doc, children, codeblocks)
        }
    }

    private class ErlangStructureItem(val dfn:AstDfn, pResult:ParserResult) extends StructureItem {
        import ElementKind._

        override
        def getName :String = dfn.getName

        override
        def getSortText :String = getName

        override
        def getHtml(formatter:HtmlFormatter) :String = {
            dfn.htmlFormat(formatter)
            formatter.getText
        }

        override
        def getElementHandle :ElementHandle = dfn

        override
        def getKind :ElementKind = dfn.getKind
        
        override
        def getModifiers :Set[Modifier] = dfn.getModifiers

        override
        def isLeaf :Boolean = dfn.getKind match {
            case MODULE | CLASS => false
            case CONSTRUCTOR | METHOD | FIELD | VARIABLE | OTHER | PARAMETER | ATTRIBUTE => true
            case _ => true
        }

        override
        def getNestedItems : List[StructureItem] = {
            val nested = dfn.bindingScope.dfns
            if (nested.size > 0) {
                val children = new ArrayList[StructureItem](nested.size)

                for (child <- nested) {
                    child.kind match {
                        case PARAMETER | VARIABLE | OTHER =>
                        case _ => children.add(new ErlangStructureItem(child, pResult))
                    }
                }

                children
            } else Collections.emptyList[StructureItem]
        }

        override
        def getPosition :Long = {
            try {
                LexUtil.tokenHierarchy(pResult) match {
                    case None => 0
                    case Some(th) => dfn.boundsOffset(th)
                }
            } catch {case ex:Exception => 0}
        }

        override
        def getEndPosition :Long = {
            try {
                LexUtil.tokenHierarchy(pResult) match {
                    case None => 0
                    case Some(th) => dfn.boundsEndOffset(th)
                }
            } catch {case ex:Exception => 0}
        }

        override
        def equals(o:Any) :Boolean = o match {
            case null => false
            case x:ErlangStructureItem if dfn.getKind == x.dfn.getKind && getName.equals(x.getName) => true
            case _ => false
        }

        override
        def hashCode :Int = {
            var hash = 7
            hash = (29 * hash) + (if (getName != null) getName.hashCode else 0)
            hash = (29 * hash) + (if (dfn.getKind != null) dfn.getKind.hashCode else 0)
            hash
        }

        override
        def toString = getName

        override
        def getCustomIcon :ImageIcon = null
    }
}

Which just travels the rootScope, fetches AstDfn's instances and wrap them to ErlangStructureItem.

The last step is register this structure analyzer to ErlangLanguage.Scala as usual:

    override
    def hasStructureScanner = true

    override
    def getStructureScanner = new ErlangStructureAnalyzer

For structure analyzer, you need also to register it in layer.xml:

    <folder name="CslPlugins">
        <folder name="text">
            <folder name="x-erlang">
                <file name="language.instance">
                    <attr name="instanceClass" stringvalue="org.netbeans.modules.erlang.editor.ErlangLanguage"/>
                </file>
                <file name="structure.instance">
                    <attr name="instanceClass" stringvalue="org.netbeans.modules.erlang.editor.ErlangStructureAnalyzer"/>
                </file>
            </folder>
        </folder>
    </folder>

Build and run, now open a .erl file, you got:

Click on the picture to enlarge it

The functions/attributes are shown in the left-side navigator window now, there are also some code-folding marks. Did you notice my preliminaty error-recover work? :-)

With lexer integrated, you've got all tokens of source file. You can code for pretty formatting, indentation, auto pair inserting, pair matching based on tokens now. But I'll try to integrate an Erlang parser first to get the basic infrastructure ready.

I can choose to use Erlang's native compiler, than via jInterface, Scala can rpc call Erlang runtime to parse source file and return an AST tree. But, Erlang's library function for parsing is not good on error-recover currently, so I choose to write an Erlang parser in Rats!. Rats! generated parser is bad on error messages and error-recover too, but I know how to improve it.

Migrating a NetBeans Schliemann based grammar definition to Rats! is almost straightforward. For me, as I'm already familiar with Rats!, it's an one day work. The preliminary Rats! definition for Erlang can be found at ParserErlang.rats. Then, I rewrote a new ParserErlang.rats according to latest Erlang spec in another half day, without the LL(k) limitation, the grammar rules now keep as close as the original spec.

To get it be used to generate a ParserErlang.java, I added corresponding ant target to build.xml, which looks like:

    <target name="rats" depends="init" description="Scanner">
        <echo message="Rebuilding token scanner... ${rats.package.dir}"/>
        <java fork="yes"
             dir="${src.dir}/${rats.package.dir}"
             classname="xtc.parser.Rats"
             classpath="${rats.jar}">
            <arg value="-in"/>
            <arg value="${src.dir}"/>
            <arg value="${rats.lexer.file}"/>
        </java>

        <echo message="Rebuilding grammar parser... ${rats.package.dir}"/>
        <java fork="yes"
             dir="${src.dir}/${rats.package.dir}"
             classname="xtc.parser.Rats"
             classpath="${rats.jar}">
            <arg value="-in"/>
            <arg value="${src.dir}"/>
            <arg value="${rats.parser.file}"/>
        </java>
    </target>

And rats related properties, such as "rats.lexer.file" and "rats.parser.file" are defined in nbproject/project.properties as:

rats.jar=${cluster}/modules/xtc.jar
rats.package.dir=org/netbeans/modules/erlang/editor/rats
rats.lexer.file=LexerErlang.rats
rats.parser.file=ParserErlang.rats

Running target "rats" will generate LexerErlang.java and ParserErlang.java, the first one is that we've used in ErlangLexer.scala, the later one is to be integrated into NetBeans' Parsing API as the parser.

Now, you should extend two Parsing API abstract classes, org.netbeans.modules.csl.spi.ParserResult and org.netbeans.modules.parsing.spi.Parser. The first one will carry result's AST Node and syntax errors that parser detected, the errors will be highlighted automatically in Editor. The second one is the bridge between NetBeans parsing task and your real parser (ParserErlang.java here)

There are tricks to do some error-recover in these two classes, by adding a "." or "end" or "}" etc to the buffered source chars when a syntax error occurred, it's called "sanitize" the source to recover the error. NetBeans' Ruby, JavaScripts supporting have some good examples of this trick. For my case, I will do error recover in Rats! definition later, so I do not use this trick currently, but I leave some "sanitize" related code there.

First, it's an ErlangParserResult.scala which extended ParserResult. The code is simple at the first phase.

ErlangParserResult.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor

import _root_.java.util.{Collections, ArrayList, List}
import org.netbeans.api.lexer.{TokenHierarchy, TokenId}
import org.netbeans.modules.csl.api.Error
import org.netbeans.modules.csl.api.OffsetRange
import org.netbeans.modules.csl.spi.ParserResult
import org.netbeans.modules.parsing.api.Snapshot
import org.netbeans.modules.erlang.editor.rats.ParserErlang
import xtc.tree.{GNode}

/**
 *
 * @author Caoyuan Deng
 */
class ErlangParserResult(parser:ErlangParser,
                         snapshot:Snapshot,
                         val rootNode:GNode,
                         val th:TokenHierarchy[_]) extends ParserResult(snapshot) {

    override
    protected def invalidate :Unit = {
        // XXX: what exactly should we do here?
    }

    override
    def getDiagnostics :List[Error] = _errors

    private var _errors = Collections.emptyList[Error]
    
    def errors = _errors
    def errors_=(errors:List[Error]) = {
        this._errors = new ArrayList[Error](errors)
    }

    var source :String = _
    
    /**
     * Return whether the source code for the parse result was "cleaned"
     * or "sanitized" (modified to reduce chance of parser errors) or not.
     * This method returns OffsetRange.NONE if the source was not sanitized,
     * otherwise returns the actual sanitized range.
     */
    var sanitizedRange = OffsetRange.NONE
    var sanitizedContents :String = _
    var sanitized :Sanitize = NONE

    var isCommentsAdded :Boolean = false
    
    /**
     * Set the range of source that was sanitized, if any.
     */
    def setSanitized(sanitized:Sanitize, sanitizedRange:OffsetRange, sanitizedContents:String) :Unit = {
        this.sanitized = sanitized
        this.sanitizedRange = sanitizedRange
        this.sanitizedContents = sanitizedContents
    }

    override
    def toString = {
        "ErlangParseResult(file=" + snapshot.getSource.getFileObject + ",rootnode=" + rootNode + ")"
    }
}

Then, the ErlangParser.scala which extended Parser:

ErlangParser.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor


import _root_.java.io.{IOException, StringReader}
import _root_.java.util.ArrayList
import _root_.java.util.Collection
import _root_.java.util.List
import _root_.java.util.ListIterator
import _root_.javax.swing.event.ChangeListener
import _root_.javax.swing.text.BadLocationException

import org.netbeans.modules.csl.api.ElementHandle
import org.netbeans.modules.csl.api.Error
import org.netbeans.modules.csl.api.OffsetRange
import org.netbeans.modules.csl.api.Severity
import org.netbeans.modules.csl.spi.DefaultError
import org.netbeans.modules.parsing.api.Snapshot
import org.netbeans.modules.parsing.api.Task
import org.netbeans.modules.parsing.spi.ParseException
import org.netbeans.modules.csl.api.EditHistory
import org.netbeans.modules.csl.spi.GsfUtilities
import org.netbeans.modules.csl.spi.ParserResult
import org.netbeans.modules.parsing.api.Source
import org.netbeans.modules.parsing.spi.Parser
import org.netbeans.modules.parsing.spi.Parser.Result
import org.netbeans.modules.parsing.spi.ParserFactory
import org.netbeans.modules.parsing.spi.SourceModificationEvent
import org.openide.filesystems.FileObject
import org.openide.util.Exceptions

import org.netbeans.api.editor.EditorRegistry
import org.netbeans.api.lexer.{TokenHierarchy, TokenId}
import org.netbeans.editor.BaseDocument
import org.netbeans.modules.editor.NbEditorUtilities

import xtc.parser.{ParseError, SemanticValue}
import xtc.tree.{GNode, Location}

import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId
import org.netbeans.modules.erlang.editor.rats.ParserErlang

/**
 *
 * @author Caoyuan Deng
 */
class ErlangParser extends Parser {

    private var lastResult :ErlangParserResult = _

    @throws(classOf[ParseException])
    override
    def parse(snapshot:Snapshot, task:Task, event:SourceModificationEvent) :Unit = {
        val context = new Context(snapshot, event)
        lastResult = parseBuffer(context, NONE)
        lastResult.errors = context.errors
    }

    @throws(classOf[ParseException])
    override
    def getResult(task:Task) :Result = {
        assert(lastResult != null, "getResult() called prior parse()") //NOI18N
        lastResult
    }

    override
    def cancel :Unit = {}

    override
    def addChangeListener(changeListener:ChangeListener) :Unit = {
        // no-op, we don't support state changes
    }

    override
    def removeChangeListener(changeListener:ChangeListener) :Unit = {
        // no-op, we don't support state changes
    }

    private def lexToAst(source:Snapshot, offset:Int) :Int = source match {
        case null => offset
        case _ => source.getEmbeddedOffset(offset)
    }

    private def astToLex(source:Snapshot, offset:Int) :Int = source match {
        case null => offset
        case _ => source.getOriginalOffset(offset)
    }

    private def sanitizeSource(context:Context, sanitizing:Sanitize) :Boolean = {
        false
    }

    private def sanitize(context:Context, sanitizing:Sanitize) :ErlangParserResult = {
        sanitizing match {
            case NEVER =>
                createParseResult(context)
            case NONE =>
                createParseResult(context)
            case _ =>
                // we are out of trick, just return as it
                createParseResult(context)
        }
    }

    protected def notifyError(context:Context, message:String, sourceName:String,
                              start:Int, lineSource:String, end:Int,
                              sanitizing:Sanitize, severity:Severity,
                              key:String, params:Object) :Unit = {

        val error = new DefaultError(key, message, null, context.fo, start, end, severity)

        params match {
            case null =>
            case x:Array[Object] => error.setParameters(x)
            case _ => error.setParameters(Array(params))
        }

        context.notifyError(error)

        if (sanitizing == NONE) {
            context.errorOffset = start
        }
    }

    protected def parseBuffer(context:Context, sanitizing:Sanitize) :ErlangParserResult = {
        var sanitizedSource = false
        var source = context.source

        sanitizing match {
            case NONE | NEVER =>
            case _ =>
                val ok = sanitizeSource(context, sanitizing)
                if (ok) {
                    assert(context.sanitizedSource != null)
                    sanitizedSource = true
                    source = context.sanitizedSource
                } else {
                    // Try next trick
                    return sanitize(context, sanitizing)
                }
        }

        if (sanitizing == NONE) {
            context.errorOffset = -1
        }

        val parser = createParser(context)

        val ignoreErrors = sanitizedSource
        var root :GNode = null
        try {
            var error :ParseError = null
            val r = parser.pS(0)
            if (r.hasValue) {
                val v = r.asInstanceOf[SemanticValue]
                root = v.value.asInstanceOf[GNode]
            } else {
                error = r.parseError
            }

            if (error != null && !ignoreErrors) {
                var start = 0
                if (error.index != -1) {
                    start = error.index
                }
                notifyError(context, error.msg, "Syntax error",
                            start, "", start,
                            sanitizing, Severity.ERROR,
                            "SYNTAX_ERROR", Array(error))

                System.err.println(error.msg)
            }

        } catch {
            case e:IOException => e.printStackTrace
            case e:IllegalArgumentException =>
                // An internal exception thrown by parser, just catch it and notify
                notifyError(context, e.getMessage, "",
                            0, "", 0,
                            sanitizing, Severity.ERROR,
                            "SYNTAX_ERROR", Array(e))
        }

        if (root != null) {
            context.sanitized = sanitizing
            context.root = root
            val r = createParseResult(context)
            r.setSanitized(context.sanitized, context.sanitizedRange, context.sanitizedContents)
            r.source = source
            r
        } else {
            sanitize(context, sanitizing)
        }
    }

    protected def createParser(context:Context) :ParserErlang = {
        val in = new StringReader(context.source)
        val fileName = if (context.fo != null) context.fo.getNameExt else ""

        val parser = new ParserErlang(in, fileName)
        context.parser = parser

        parser
    }

    private def createParseResult(context:Context) :ErlangParserResult = {
        new ErlangParserResult(this, context.snapshot, context.root, context.th)
    }

    /** Parsing context */
    class Context(val snapshot:Snapshot, event:SourceModificationEvent) {
        val errors :List[Error] = new ArrayList[Error]

        var source :String = ErlangParser.asString(snapshot.getText)
        var caretOffset :Int = GsfUtilities.getLastKnownCaretOffset(snapshot, event)

        var root :GNode = _
        var th :TokenHierarchy[_] = _
        var parser :ParserErlang = _
        var errorOffset :Int = _
        var sanitizedSource :String = _
        var sanitizedRange :OffsetRange = OffsetRange.NONE
        var sanitizedContents :String = _
        var sanitized :Sanitize = NONE

        def notifyError(error:Error) = errors.add(error)

        def fo = snapshot.getSource.getFileObject

        override
        def toString = "ErlangParser.Context(" + fo + ")" // NOI18N

    }
}

object ErlangParser {
    def asString(sequence:CharSequence) :String = sequence match {
        case s:String => s
        case _ => sequence.toString
    }

    def sourceUri(source:Source) :String = source.getFileObject match {
        case null => "fileless" //NOI18N
        case f => f.getNameExt
    }
}

/** Attempts to sanitize the input buffer */
sealed case class Sanitize
/** Only parse the current file accurately, don't try heuristics */
case object NEVER extends Sanitize
/** Perform no sanitization */
case object NONE extends Sanitize

The major tasks of these two classes are:

• Parsing the source buffer which is passed automatically by the framework when source is modified or just opened, the source text can be got from snapshot.getText. The parsing result used to be an AST tree, you can do semantic/structure analysis on AST tree to get more detailed information later, but as the first step, I just pass/keep AST tree in ErlangParserResult for later usage.
• Notifying the errors to framework. This can be done by store the parsing errors to ParserResult, and implemented getDiagnostics to return the error.

The last step is register the ErlangParser in ErlangLanguage.scala in one line code:

override def getParser = new ErlangParser

Now, open a .erl file, if there is syntax error, the editor will indicate and highlight it now.

My next step is to improve the Rats! definition, add error recover etc.

I'm going to create new Erlang Editor module in Scala, the module project is under http://hg.netbeans.org/main/contrib/file/0caa5d009839/erlang.editor/ which I just committed in.

The first step is to get Scala source file mixed in Java based NetBeans source/project tree. After define project's manifest,mf, we'll create a special build.xml for this project under erlang.editor directory:

build.xml

<?xml version="1.0" encoding="UTF-8"?>
<project name="contrib/erlang.editor" default="netbeans" basedir=".">
    <import file="../../nbbuild/templates/projectized.xml"/>
    <import file="scala-build.xml"/>

    <!-- special jar target for csl -->
    <target name="jar" depends="init,compile,jar-prep" unless="is.jar.uptodate">
        <taskdef name="csljar" classname="org.netbeans.modules.csl.CslJar" classpath="${nb_all}/csl.api/anttask/build/cslanttask.jar:${nb_all}/nbbuild/nbantext.jar"/>
        <csljar jarfile="${cluster}/${module.jar}" compress="${build.package.compress}" index="${build.package.index}" manifest="${manifest.mf}" stamp="${cluster}/.lastModified">
            <fileset dir="${build.classes.dir}"/>
        </csljar>
    </target>

    <target name="compile" depends="init,projectized-common.compile,scala-compile"/>
    <target name="do-test-build" depends="init,test-init,projectized-common.do-test-build"/>

    <target name="rats" depends="init" description="Scanner">
        <echo message="Rebuilding token scanner... ${rats.package.dir}"/>
        <java fork="yes"
             dir="${src.dir}/${rats.package.dir}"
             classname="xtc.parser.Rats"
             classpath="${rats.jar}">
            <arg value="-in"/>
            <arg value="${src.dir}"/>
            <arg value="${rats.lexer.file}"/>
        </java>
    </target>
    
</project>

This above build.xml imported NetBeans' main template ant xml "../../nbbuild/templates/projectized.xml" and a special scala-build.xml file, which defines scalac and javac task for mixed Scala/Java module for NetBeans. You can ignore the "rats" target if you has no plan to write language's lexer/parser in Rats! parser generator. scala-build.xml looks like:

scala-build.xml

<?xml version="1.0" encoding="UTF-8"?>
<project name="scala-module" default="netbeans" basedir=".">
    <import file="../../nbbuild/templates/projectized.xml"/>
    
    <target name="scala-taskdef" depends="init">
        <echo message="Compiling scala sources via ${scala.library}, ${scala.compiler}"/>
        <taskdef resource="scala/tools/ant/antlib.xml">
            <classpath>
                <pathelement location="${scala.library}"/>
                <pathelement location="${scala.compiler}"/>
            </classpath>
        </taskdef>
    </target>

    <property name="jar-excludes" value="**/*.java,**/*.form,**/package.html,**/doc-files/,**/*.scala"/>
    
    <target name="scala-compile" depends="init,up-to-date,scala-taskdef" unless="is.jar.uptodate">
        <!-- javac's classpath should include scala.library and all these paths of "cp" -->
        <path id="javac.cp">
            <pathelement path="${scala.libs}"/>
            <pathelement path="${module.classpath}"/>
            <pathelement path="${cp.extra}"/>
        </path>
        <!-- scalac will check class dependencies deeply, so we can not rely on public package only which is refed by ${module.classpath} -->
        <path id="scalac.cp">
            <pathelement path="${scala.libs}"/>
            <pathelement path="${module.run.classpath}"/>
            <pathelement path="${cp.extra}"/>
        </path>
        <mkdir dir="${build.classes.dir}"/>
        <depend srcdir="${src.dir}" destdir="${build.classes.dir}" cache="build/depcache">
            <classpath refid="scalac.cp"/>
        </depend>
        <!-- scalac -->
        <scalac srcdir="${src.dir}" destdir="${build.classes.dir}" encoding="UTF-8" target="jvm-${javac.target}">
            <classpath refid="scalac.cp"/>
        </scalac>
        <!-- javac -->
        <nb-javac srcdir="${src.dir}" destdir="${build.classes.dir}" debug="${build.compiler.debug}" debuglevel="${build.compiler.debuglevel}" encoding="UTF-8"
                deprecation="${build.compiler.deprecation}" optimize="${build.compiler.optimize}" source="${javac.source}" target="${javac.target}" includeantruntime="false">
            <classpath refid="javac.cp"/>
            <compilerarg line="${javac.compilerargs}"/>
            <processorpath refid="processor.cp"/>
        </nb-javac>
        <!-- Sanity check: -->
        <pathconvert pathsep=":" property="class.files.in.src">
            <path>
                <fileset dir="${src.dir}">
                    <include name="**/*.class"/>
                </fileset>
            </path>
        </pathconvert>
        <fail>
            <condition>
                <not>
                    <equals arg1="${class.files.in.src}" arg2=""/>
                </not>
            </condition>
            You have stray *.class files in ${src.dir} which you must remove.
            Probably you failed to clean your sources before updating them.
        </fail>
        <!-- OK, continue: -->
        <copy todir="${build.classes.dir}">
            <fileset dir="${src.dir}" excludes="${jar-excludes}"/>
        </copy>
    </target>

    <target name="do-test-build" depends="projectized-common.do-test-build">
        <scalac srcdir="${test.unit.src.dir}" destdir="${build.test.unit.classes.dir}" excludes="${test.excludes}"
               encoding="UTF-8">
            <classpath refid="test.unit.cp"/>
        </scalac>
    </target>
</project>

You need also to set the project's module dependencies on:

• org.netbeans.libs.scala
• org.netbeans.modules.csl.api
• org.netbeans.modules.lexer
• org.netbeans.modules.parsing.api
• org.openide.filesystems

And, the nbproject/project.properties which defines some important project building properties, such as:

javac.compilerargs=-Xlint:unchecked
javac.source=1.5
nbm.homepage=http://wiki.netbeans.org/Erlang

scala.library=${cluster}/modules/ext/scala-library-2.7.3.jar
scala.compiler=${cluster}/modules/ext/scala-compiler-2.7.3.jar
scala.libs=\
   ${scala.library}:\
   ${scala.compiler}

You can write NetBeans' module in Scala now.

>>> Updated Feb 8:

Code fixed to avoid an infinite cycled scanning when source file size > 16k

===

The minim supporting is to integrate a language lexer to NetBeans's language support framework. As NetBeans 7.0, there is a new effort for common language supporting, which is called CSL (Common Scripting Language). Don't be confused by this module's name, it not only for scripting language, actually, Java support has been migrated to CSL in 7.0. There are discussions on a better name. CSL is forked and created on GSF (Generic Scripting Framework) as a GSF's new variant that is based on new Parsing & Indexing API.

Since I'm going to write an Erlang lexer in Rats! generator parser, I need to add dependency on rats run-time libs first, rats run-time module is under contrib/xtc, which I patched to support end position of each production.

Then, you have to tell the project where to find the rats! libs, this can be done by adding following properties to nbproject/project.properties, now this nbproject/project.properties looks like:

nbproject/project.properties

javac.compilerargs=-Xlint:unchecked
javac.source=1.5
nbm.homepage=http://wiki.netbeans.org/Erlang

scala.library=${cluster}/modules/ext/scala-library-2.7.3.jar
scala.compiler=${cluster}/modules/ext/scala-compiler-2.7.3.jar
scala.libs=\
   ${scala.library}:\
   ${scala.compiler}

rats.jar=${cluster}/modules/xtc.jar
rats.package.dir=org/netbeans/modules/erlang/editor/rats
rats.lexer.file=LexerErlang.rats

All Rats! definitions of Erlang token can be found at http://hg.netbeans.org/main/contrib/file/tip/erlang.editor/src/org/netbeans/modules/erlang/editor/rats/. Don't ask me how to write Rats! rules for languages, you should get these information from Rats! web site. Or, you can integrate other types of lexer generated by other lexer generator, the examples can be found in NetBeans' other languages supporting modules.

The Erlang's lexer will be generated via "rats.lexer.file=LexerErlang.rats", which is the entry point of all defined rules for Erlang tokens. Run "rats" target will generate a LexerErlang.java file which is the lexer class that will be used to create Erlang tokens from Erlang source files.

Now, we should integrate this lexer class to NetBeans' lexer engine, this is done by two Scala files:

ErlangLexer.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2008 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor.lexer

import _root_.java.io.IOException
import _root_.java.io.Reader
import org.netbeans.api.lexer.{Token, TokenId}
import org.netbeans.modules.erlang.editor.rats.LexerErlang
import org.netbeans.spi.lexer.Lexer
import org.netbeans.spi.lexer.LexerInput
import org.netbeans.spi.lexer.LexerRestartInfo
import org.netbeans.spi.lexer.TokenFactory
import xtc.parser.Result
import xtc.tree.GNode
import xtc.util.Pair

import scala.collection.mutable.ArrayBuffer

import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId._

/**
 *
 * @author Caoyuan Deng
 */
object ErlangLexer {
    /** @Note:
     * ErlangLexer class is not Reentrant safe, it seems when source size is large than 16 * 1024,
     * there will be more than one input are used, which causes the offset states, such as readed 
     * token length, offset etc in these inputs conflict?. Anyway it's safe to create a new one always.
     */
    def create(info:LexerRestartInfo[TokenId]) = new ErlangLexer(info)
}

class ErlangLexer(info:LexerRestartInfo[TokenId]) extends Lexer[TokenId] {
    /** @Note:
     * it seems input at this time is empty, so we can not do scanning here.
     * input will be filled in chars when call nextToken
     */

    var input :LexerInput = info.input
    var tokenFactory :TokenFactory[TokenId] = info.tokenFactory
    var lexerInputReader :LexerInputReader = new LexerInputReader(input)
    
    val tokenStream = new ArrayBuffer[TokenInfo]
    // * tokenStream.elements always return a new iterator, which point the first
    // * item, so we should have a global one.
    var tokenStreamItr :Iterator[TokenInfo]  = tokenStream.elements
    var lookahead :Int = 0

    override
    def release = {}

    override
    def state :Object = null

    override
    def nextToken :Token[TokenId] = {
        // * In case of embedded tokens, there may be tokens that had been scanned
        // * but not taken yet, check first
        if (!tokenStreamItr.hasNext) {
            tokenStream.clear
            scanTokens
            tokenStreamItr = tokenStream.elements

            /**
             * @Bug of LexerInput.backup(int) ?
             * backup(0) will cause input.readLength() increase 1
             */
            lookahead = input.readLength
            if (lookahead > 0) {
                // * backup all, we will re-read from begin to create token at following step
                input.backup(lookahead)
            } else {
                return null
            }
        }

        if (tokenStreamItr.hasNext) {
            val tokenInfo = tokenStreamItr.next

            if (tokenInfo.length == 0) {
                // * EOF
                return null
            }

            // * read token's chars according to tokenInfo.length
            var i = 0
            while (i < tokenInfo.length) {
                input.read
                i += 1
            }

            // * see if needs to lookahead, if true, perform it
            lookahead -= tokenInfo.length
            // * to cheat incremently lexer, we needs to lookahead one more char when
            // * tokenStream.size() > 1 (batched tokens that are not context free),
            // * so, when modification happens extractly behind latest token, will
            // * force lexer relexer from the 1st token of tokenStream
            val lookahead1 = if (tokenStream.size > 1) lookahead + 1 else lookahead
            if (lookahead1 > 0) {
                var i = 0
                while (i < lookahead1) {
                    input.read
                    i += 1
                }
                input.backup(lookahead1)
            }

            val tokenLength = input.readLength
            createToken(tokenInfo.id, tokenLength)
        } else {
            assert(false, "unrecognized input" + input.read)
            null
        }
    }

    def createToken(id:TokenId, length:Int) :Token[TokenId] = id.asInstanceOf[ErlangTokenId].fixedText match {
        case null => tokenFactory.createToken(id, length)
        case fixedText => tokenFactory.getFlyweightToken(id, fixedText)
    }

    def scanTokens :Result = {
        /**
         * We cannot keep an instance scope lexer, since lexer (sub-class of ParserBase)
         * has internal states which keep the read-in chars, index and others, it really
         * difficult to handle.
         */
        val scanner = new LexerErlang(lexerInputReader, "")
        try {
            // * just scan from position 0, incrmental lexer engine will handle start char in lexerInputReader
            val r = scanner.pToken(0)
            if (r.hasValue) {
                val node = r.semanticValue.asInstanceOf[GNode]
                flattenToTokenStream(node)
                r
            } else {
                System.err.println(r.parseError.msg)
                null
            }
        } catch {
            case e:Exception =>
                System.err.println(e.getMessage)
                null
        }
    }

    def flattenToTokenStream(node:GNode) :Unit = {
        val l = node.size
        if (l == 0) {
            /** @Note:
             * When node.size == 0, it's a void node. This should be limited to
             * EOF when you define lexical rats.
             *
             * And in Rats!, EOF is !_, the input.readLength() will return 0
             */      
            assert(input.readLength == 0,
                   "This generic node: " + node.getName +
                   " is a void node, this should happen only on EOF. Check you rats file.")

            val tokenInfo = new TokenInfo(0, null)
            tokenStream += tokenInfo
            return
        }
        
        var i = 0
        while (i < l) {
            node.get(i) match {
                case null =>
                    // * child may be null
                case child:GNode =>
                    flattenToTokenStream(child)
                case child:Pair[_] =>
                    assert(false, "Pair:" + child + " to be process, do you add 'flatten' option on grammar file?")
                case child:String =>
                    val length = child.length
                    val id = ErlangTokenId.valueOf(node.getName) match {
                        case None => ErlangTokenId.IGNORED
                        case Some(v) => v.asInstanceOf[TokenId]
                    }
          
                    val tokenInfo = new TokenInfo(length, id)
                    tokenStream += tokenInfo
                case child =>
                    println("To be process: " + child)
            }
            i += 1
        }
    }

    /**
     * Hacking for xtc.parser.ParserBase of Rats! which use java.io.Reader
     * as the chars input, but uses only {@link java.io.Reader#read()} of all methods in
     * {@link xtc.parser.ParserBase#character(int)}
     */
    class LexerInputReader(input:LexerInput) extends Reader {
        override
        def read :Int = input.read match {
            case LexerInput.EOF => -1
            case c => c
        }

        override
        def read(cbuf:Array[Char], off:Int, len:Int) :Int = {
            throw new UnsupportedOperationException("Not supported yet.")
        }

        override
        def close = {}
    }

    class TokenInfo(val length:Int, val id:TokenId) {
        override
        def toString = "(id=" + id + ", length=" + length + ")"
    }
}

ErlangTokenId

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor.lexer

import _root_.java.util.Collection
import _root_.java.util.Collections
import _root_.java.util.HashMap
import _root_.java.util.HashSet
import _root_.java.util.Map
import _root_.java.util.Arrays

import org.netbeans.api.lexer.InputAttributes
import org.netbeans.api.lexer.Language
import org.netbeans.api.lexer.LanguagePath
import org.netbeans.api.lexer.Token
import org.netbeans.api.lexer.TokenId
import org.netbeans.spi.lexer.LanguageEmbedding
import org.netbeans.spi.lexer.LanguageHierarchy
import org.netbeans.spi.lexer.Lexer
import org.netbeans.spi.lexer.LexerRestartInfo

/**
 * 
 * @author Caoyuan Deng
 */
object ErlangTokenId extends Enumeration {
    // Let type of enum's value the same as enum itself
    type ErlangTokenId = V

    // Extends Enumeration.Val to get custom enumeration value
    class V(val name:String, val fixedText:String, val primaryCategory:String) extends Val(name) with TokenId {
        override
        def ordinal = id
    }
    object V {
        def apply(name:String, fixedText:String, primaryCategory:String) = new V(name, fixedText, primaryCategory)
    }
  
    val IGNORED = V("IGNORED", null, "ingore")
    val Error = V("Error", null, "error")

    // --- Spaces and comments
    val Ws = V("Ws", null, "whitespace")
    val Nl = V("Nl", null, "whitespace")
    val LineComment = V("LineComment", null, "comment")
    val CommentTag = V("CommentTag", null, "comment")
    val CommentData = V("CommentData", null, "comment")

    // --- Literals
    val IntegerLiteral = V("IntegerLiteral", null, "number")
    val FloatingPointLiteral = V("FloatingPointLiteral", null, "number")
    val CharacterLiteral = V("CharacterLiteral", null, "char")
    val StringLiteral = V("StringLiteral", null, "string")

    // --- Keywords
    val Andalso = V("Andalso", "andalso", "keyword")
    val After = V("After", "after", "keyword")
    val And = V("And", "and", "keyword")
    val Band = V("Band", "band", "keyword")
    val Begin = V("Begin", "begin", "keyword")
    val Bnot = V("Bnot", "bnot", "keyword")
    val Bor = V("Bor", "bor", "keyword")
    val Bsr = V("Bsr", "bsr", "keyword")
    val Bxor = V("Bxor", "bxor", "keyword")
    val Case = V("Case", "case", "keyword")
    val Catch = V("Catch", "catch", "keyword")
    val Cond = V("Cond", "cond", "keyword")
    val Div = V("Div", "div", "keyword")
    val End = V("End", "end", "keyword")
    val Fun = V("Fun", "fun", "keyword")
    val If = V("If", "if", "keyword")
    val Not = V("Not", "not", "keyword")
    val Of = V("Of", "of", "keyword")
    val Orelse = V("Orelse", "orelse", "keyword")
    val Or = V("Or", "or", "keyword")
    val Query = V("Query", "query", "keyword")
    val Receive = V("Receive", "receive", "keyword")
    val Rem = V("Rem", "rem", "keyword")
    val Try = V("Try", "try", "keyword")
    val Spec = V("Spec", "spec", "keyword")
    val When = V("When", "when", "keyword")
    val Xor = V("Xor", "xor", "keyword")

    // --- Identifiers
    val Macro = V("Macro", null, "identifier")
    val Atom = V("Atom", null, "identifier")
    val Var = V("Var", null, "identifier")
    val Rec = V("Rec", null, "identifier")

    // --- Stop
    val Stop = V("Stop", ".", "separator")

    // --- Symbols
    val LParen = V("LParen", "(", "separator")
    val RParen = V("RParan", ")", "separator")
    val LBrace = V("LBrace", "{", "separator")
    val RBrace = V("RBrace", "}", "separator")
    val LBracket = V("LBracket", "[", "separator")
    val RBracket = V("RBracket", "]", "separator")
    val Comma = V("Comma", ",", "separator")
    val Dot = V("Dot", ".", "separator")
    val Semicolon = V("Semicolon", ";", "separator")
    val DBar = V("DBar", "||", "separator")
    val Bar = V("Bar", "|",  "separator")
    val Question = V("Question", "?","separator")
    val DLt = V("DLt", "<<", "separator")
    val LArrow = V("LArrow", "<-", "separator")
    val Lt = V("Lt", "<", "separator")
    val DGt = V("DGt", ">>", "separator")
    val Ge = V("Ge", ">=", "separator")
    val Gt = V("Gt", ">", "separator")
    val ColonMinus = V("ColonMinus", ":-", "separator")
    val DColon = V("DColon", "::", "separator")
    val Colon = V("Colon", ":", "separator")
    val Hash = V("Hash", "#", "separator")
    val DPlus = V("DPlus", "++", "separator")
    val Plus = V("Plus", "+", "separator")
    val DMinus = V("DMinus", "--", "separator")
    val RArrow = V("RArrow", "->", "separator")
    val Minus = V("Minus", "-", "separator")
    val Star = V("Star", "*", "separator")
    val Ne = V("Ne", "/=", "separator")
    val Slash = V("Slash", "/", "separator")
    val EEq = V("EEq", "=:=", "separator")
    val ENe = V("ENe", "=/=", "separator")
    val DEq = V("DEq", "==", "separator")
    val Le = V("le", "=<", "separator")
    val Eq = V("Eq", "=", "separator")
    val Exclamation = V("Exclamation", "!", "separator")

  
    /**
     * MIME type for Erlang. Don't change this without also consulting the various XML files
     * that cannot reference this value directly.
     */
    val ERLANG_MIME_TYPE = "text/x-erlang"; // NOI18N

    // * should use "val" instead of "def" here to get a singleton language val, which  
    // * will be used to identity the token's language by "==" comparasion by other classes.
    // * Be aware of the init order! to get createTokenIds gathers all TokenIds, should
    // * be put after all token id val definition
    val language = new LanguageHierarchy[TokenId] {
        protected def mimeType = ERLANG_MIME_TYPE

        protected def createTokenIds :Collection[TokenId] = {
            val ids = new HashSet[TokenId]
            elements.foreach{ids add _.asInstanceOf[TokenId]}
            ids
        }
    
        protected def createLexer(info:LexerRestartInfo[TokenId]) :Lexer[TokenId] = ErlangLexer.create(info)

        override
        protected def createTokenCategories :Map[String, Collection[TokenId]] = {
            val cats = new HashMap[String, Collection[TokenId]]
            cats
        }

        override
        protected def embedding(token:Token[TokenId], languagePath:LanguagePath, inputAttributes:InputAttributes) = {
            null // No embedding
        }
    }.language

}

ErlangTokenId implemented org.netbeans.api.lexer.TokenId, and defined all Erlang token's Ids, with each name, id, fixedText and primaryCategory.

ErlangLexer.scala is the bridge between LexerErlang.java and NetBeans' lexer engine. NetBeans' lexer engine is an incremental engine, which will automatically handle the positions when you insert/modify a char, wrap the sanitary buffer in a LexerInput and pass it to lexer. I have carefully degined ErlangLexer.scala to get these benefits, you can use this file for other languages too, if you have understood how I write rats rules for tokens.

Now, you should implemented an org.netbeans.modules.csl.spi.DefaultLanguageConfig, which register all services for your language, such as: CodeCompletionHandler, DeclarationFinder, Formatter, IndexSearcher, InstantRenamer, KeystrokeHandler, OccurrencesFinder, SemanticAnalyzer, StructureScanner etc. For the first step, we only implemented a minim supporting for Erlang, which actually is only a lexer to get Erlang tokens and highlight them. So, our implementation is fairly simple:

ErlangLanguage.scala

/*
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
 *
 * Copyright 1997-2007 Sun Microsystems, Inc. All rights reserved.
 *
 * The contents of this file are subject to the terms of either the GNU
 * General Public License Version 2 only ("GPL") or the Common
 * Development and Distribution License("CDDL") (collectively, the
 * "License"). You may not use this file except in compliance with the
 * License. You can obtain a copy of the License at
 * http://www.netbeans.org/cddl-gplv2.html
 * or nbbuild/licenses/CDDL-GPL-2-CP. See the License for the
 * specific language governing permissions and limitations under the
 * License.  When distributing the software, include this License Header
 * Notice in each file and include the License file at
 * nbbuild/licenses/CDDL-GPL-2-CP.  Sun designates this
 * particular file as subject to the "Classpath" exception as provided
 * by Sun in the GPL Version 2 section of the License file that
 * accompanied this code. If applicable, add the following below the
 * License Header, with the fields enclosed by brackets [] replaced by
 * your own identifying information:
 * "Portions Copyrighted [year] [name of copyright owner]"
 *
 * Contributor(s):
 *
 * The Original Software is NetBeans. The Initial Developer of the Original
 * Software is Sun Microsystems, Inc. Portions Copyright 1997-2006 Sun
 * Microsystems, Inc. All Rights Reserved.
 *
 * If you wish your version of this file to be governed by only the CDDL
 * or only the GPL Version 2, indicate your decision by adding
 * "[Contributor] elects to include this software in this distribution
 * under the [CDDL or GPL Version 2] license." If you do not indicate a
 * single choice of license, a recipient has the option to distribute
 * your version of this file under either the CDDL, the GPL Version 2 or
 * to extend the choice of license to its licensees as provided above.
 * However, if you add GPL Version 2 code and therefore, elected the GPL
 * Version 2 license, then the option applies only if the new code is
 * made subject to such option by the copyright holder.
 */
package org.netbeans.modules.erlang.editor

import _root_.java.io.File
import _root_.java.util.Collection
import _root_.java.util.Collections
import _root_.java.util.HashMap
import _root_.java.util.Map
import _root_.java.util.Set
import org.netbeans.api.lexer.Language;
import org.netbeans.modules.csl.api.CodeCompletionHandler
import org.netbeans.modules.csl.api.DeclarationFinder
import org.netbeans.modules.csl.api.Formatter
import org.netbeans.modules.csl.api.IndexSearcher
import org.netbeans.modules.csl.api.InstantRenamer
import org.netbeans.modules.csl.api.KeystrokeHandler
import org.netbeans.modules.csl.api.OccurrencesFinder
import org.netbeans.modules.csl.api.SemanticAnalyzer
import org.netbeans.modules.csl.api.StructureScanner
import org.netbeans.modules.csl.spi.DefaultLanguageConfig
import org.netbeans.modules.parsing.spi.Parser
import org.netbeans.modules.parsing.spi.indexing.EmbeddingIndexerFactory
import org.openide.filesystems.FileObject
import org.openide.filesystems.FileUtil
import org.netbeans.modules.erlang.editor.lexer.ErlangTokenId

/*
 * Language/lexing configuration for Erlang
 *
 * @author Caoyuan Deng
 */
class ErlangLanguage extends DefaultLanguageConfig {

  override
  def getLexerLanguage = ErlangTokenId.language
    
  override
  def getDisplayName : String =  "Erlang"
    
  override
  def getPreferredExtension : String = {
    "erl" // NOI18N
  }    
}

The final step is to register ErlangLanguage and fontColor.xml, erlangResolver.xml etc in layer.xml for color highlights, mime resolver and language icon. All these necessary resource files are under: http://hg.netbeans.org/main/contrib/file/0caa5d009839/erlang.editor/src/org/netbeans/modules/erlang/editor/resources/

layer.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE filesystem PUBLIC "-//NetBeans//DTD Filesystem 1.1//EN" "http://www.netbeans.org/dtds/filesystem-1_1.dtd">
<filesystem>
    <folder name="Services">
        <folder name="MIMEResolver">
            <file name="Erlang.xml" url="erlangResolver.xml">
                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.resources.Bundle"/>
                <attr name="position" intvalue="1005"/>
            </file>
        </folder>
    </folder>

    <folder name="Editors">
      <!-- Reference binding color themes are under module: main/defaults/src/org/netbeans/modules/defaults -->
      <!-- color theme for nbeditor-settings-ColoringType -->
        <folder name="FontsColors">
            <folder name="Twilight">
                <folder name="Defaults">
                    <file name="org-netbeans-modules-defaults-highlight-colorings.xml" url="Twilight/editor.xml">
                        <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.defaults.Bundle"/>
                        <attr name="nbeditor-settings-ColoringType" stringvalue="highlight"/>
                    </file>
                    <file name="org-netbeans-modules-defaults-token-colorings.xml" url="Twilight/defaults.xml">
                        <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.defaults.Bundle"/>
                    </file>
                </folder>
            </folder>
            <folder name="EmacsStandard">
                <folder name="Defaults">
                    <file name="org-netbeans-modules-defaults-token-colorings.xml" url="EmacsStandard/defaults.xml">
                        <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.defaults.Bundle"/>
                    </file>
                </folder>
            </folder>
        </folder>

        <folder name="text">
            <folder name="x-erlang">
                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                <file name="language.instance">
                    <attr name="instanceCreate" methodvalue="org.netbeans.modules.erlang.editor.lexer.ErlangTokenId.language"/>
                    <attr name="instanceOf" stringvalue="org.netbeans.api.lexer.Language"/>
                </file>
                
                <!-- TODO - this should not be necessary; I'm doing this now to work around
                    bugs in color initialization -->
                <folder name="FontsColors">
                    <folder name="NetBeans">
                        <folder name="Defaults">
                            <file name="coloring.xml" url="fontsColors.xml">
                                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                            </file>
                        </folder>
                    </folder>
                    <folder name="Twilight">
                        <folder name="Defaults">
                            <file name="coloring.xml" url="Twilight/fontsColors.xml">
                                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                            </file>
                        </folder>
                    </folder>
                    <folder name="EmacsStandard">
                        <folder name="Defaults">
                            <file name="coloring.xml" url="EmacsStandard/fontsColors.xml">
                                <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                            </file>
                        </folder>
                    </folder>
                </folder>

                <folder name="CodeTemplates">
                    <folder name="Defaults">
                        <file name="codeTemplates.xml" url="codeTemplates.xml">
                            <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.erlang.editor.Bundle"/>
                        </file>
                    </folder>
                </folder>
                <folder name="Keybindings">
                    <folder name="NetBeans">
                        <folder name="Defaults">
                            <file name="org-netbeans-modules-erlang-editor-keybindings.xml" url="keyBindings.xml"/>
                        </folder>
                    </folder>
                </folder>
            </folder>
        </folder>
    </folder>

    <folder name="CslPlugins">
        <folder name="text">
            <folder name="x-erlang">
                <file name="language.instance">
                    <attr name="instanceClass" stringvalue="org.netbeans.modules.erlang.editor.ErlangLanguage"/>
                </file>
            <!--file name="structure.instance">
               <attr name="instanceClass" stringvalue="org.netbeans.modules.scala.editing.ScalaStructureAnalyzer"/>
            </file-->
            </folder>
        </folder>
    </folder>
    
    <folder name="Loaders">
        <folder name="text">
            <folder name="x-erlang">
                <attr name="SystemFileSystem.icon" urlvalue="nbresloc:/org/netbeans/modules/erlang/editor/resources/Erlang.png"/>
                <attr name="iconBase" stringvalue="org/netbeans/modules/erlang/editor/resources/Erlang.png"/>
                <folder name="Actions">
                    <file name="OpenAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.OpenAction"/>
                        <attr name="position" intvalue="100"/>
                    </file>
                    <file name="Separator1.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="200"/>
                    </file>
                    <file name="CutAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.CutAction"/>
                        <attr name="position" intvalue="300"/>
                    </file>
                    <file name="CopyAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.CopyAction"/>
                        <attr name="position" intvalue="400"/>
                    </file>
                    <file name="PasteAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.PasteAction"/>
                        <attr name="position" intvalue="500"/>
                    </file>
                    <file name="Separator2.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="600"/>
                    </file>
                    <file name="NewAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.NewAction"/>
                        <attr name="position" intvalue="700"/>
                    </file>
                    <file name="DeleteAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.DeleteAction"/>
                        <attr name="position" intvalue="800"/>
                    </file>
                    <file name="RenameAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.RenameAction"/>
                        <attr name="position" intvalue="900"/>
                    </file>
                    <file name="Separator3.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="1000"/>
                    </file>
                    <file name="SaveAsTemplateAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.SaveAsTemplateAction"/>
                        <attr name="position" intvalue="1100"/>
                    </file>
                    <file name="Separator4.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="1200"/>
                    </file>
                    <file name="FileSystemAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.FileSystemAction"/>
                        <attr name="position" intvalue="1300"/>
                    </file>
                    <file name="Separator5.instance">
                        <attr name="instanceClass" stringvalue="javax.swing.JSeparator"/>
                        <attr name="position" intvalue="1400"/>
                    </file>
                    <file name="ToolsAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.ToolsAction"/>
                        <attr name="position" intvalue="1500"/>
                    </file>
                    <file name="PropertiesAction.instance">
                        <attr name="instanceClass" stringvalue="org.openide.actions.PropertiesAction"/>
                        <attr name="position" intvalue="1600"/>
                    </file>
                </folder>
            </folder>
        </folder>
    </folder>
</filesystem>

Now build your new module, lunch it, and open a .erl file, you get:

Click on the picture to enlarge it

ErlyBird (previous plugin) was written in Java, and based on NetBeans' Generic Languages Framework ( Project Schliemann). As NetBeans 7.0 introducing new  Parsing API, I'm going to migrate Erlang plugin for NetBeans to it, and rewrite in Scala.

Defining a Scala/Java mixed module project in NetBeans' trunk tree is a bit trick, but, since it's plain ant-based project, you can always take over it by writing/modifying build.xml. Challenges are actually from integrating Scala's Class/Object/Trait to an already existed framework. I met this kind of challenge quickly when I tried to implement an enum like class which should implement a Java interface org.netbeans.api.lexer.TokenId?

The signature of TokenId? is:

public interface TokenId {
    String name();
    int ordinal();
    String primaryCategory();
}

From the signature, you can get the hint to implement it as a Java enum. That's true and straightforward in Java. But how about in Scala?

There is an abstract class  Enumeration in Scala's library. The example code shows how to use it by extending it as a singleton Object. But for my case, I have to implement name() and ordinal() methods of TokenId? in the meantime, which brings some inconveniences for get the name() method automatically/magically for each enumeration value.

A quick google search gave some discussions about it, for example, a method reflection and invocation may be a choice to get name() simply implemented. I tried it, but finally dropped, because you cannot guarantee the condition of call stack of name(), it may happen to be called in this Object's own method, which then, may bring infinite cycled calls.

My final code is like:

package org.netbeans.modules.erlang.editor.lexer

object ErlangTokenId extends Enumeration {
  type ErlangTokenId = V

  // Extends Enumeration's inner class Val to get custom enumeration value
  class V(val name:String, val fixedText:String, val primaryCategory:String) extends Val(name) with TokenId {
    override
    def ordinal = id
  }
  object V {
    def apply(name:String, fixedText:String, primaryCategory:String) = new V(name, fixedText, primaryCategory)
  }
  
  val IGNORED = V("IGNORED", null, "ingore")
  val Error = V("Error", null, "error")

  // --- Spaces and comments
  val Ws = V("Ws", null, "whitespace")
  val Nl = V("Nl", null, "whitespace")
  val LineComment = V("LineComment", null, "comment")
  val CommentTag = V("CommentTag", null, "comment")
  val CommentData = V("CommentData", null, "comment")

  // --- Literals
  val IntegerLiteral = V("IntegerLiteral", null, "number")
  val FloatingPointLiteral = V("FloatingPointLiteral", null, "number")
  val CharacterLiteral = V("CharacterLiteral", null, "char")
  val StringLiteral = V("StringLiteral", null, "string")

  // --- Keywords
  val Andalso = V("Andalso", "andalso", "keyword")
  val After = V("After", "after", "keyword")
  val And = V("And", "and", "keyword")
  val Band = V("Band", "band", "keyword")
  val Begin = V("Begin", "begin", "keyword")
  val Bnot = V("Bnot", "bnot", "keyword")
  val Bor = V("Bor", "bor", "keyword")
  val Bsr = V("Bsr", "bsr", "keyword")
  val Bxor = V("Bxor", "bxor", "keyword")
  val Case = V("Case", "case", "keyword")
  val Catch = V("Catch", "catch", "keyword")
  val Cond = V("Cond", "cond", "keyword")
  val Div = V("Div", "div", "keyword")
  val End = V("End", "end", "keyword")
  val Fun = V("Fun", "fun", "keyword")
  val If = V("If", "if", "keyword")
  val Not = V("Not", "not", "keyword")
  val Of = V("Of", "of", "keyword")
  val Orelse = V("Orelse", "orelse", "keyword")
  val Or = V("Or", "or", "keyword")
  val Query = V("Query", "query", "keyword")
  val Receive = V("Receive", "receive", "keyword")
  val Rem = V("Rem", "rem", "keyword")
  val Try = V("Try", "try", "keyword")
  val When = V("When", "when", "keyword")
  val Xor = V("Xor", "xor", "keyword")

  // --- Identifiers
  val Atom = V("Atom", null, "identifier")
  val Var = V("Var", null, "identifier")

  // --- Symbols
  val LParen = V("LParen", "(", "separator")
  val RParen = V("RParan", ")", "separator")
  val LBrace = V("LBrace", "{", "separator")
  val RBrace = V("RBrace", "}", "separator")
  val LBracket = V("LBracket", "[", "separator")
  val RBracket = V("RBracket", "]", "separator")
  val Comma = V("Comma", ",", "separator")
  val Dot = V("Dot", ".", "separator")
  val Semicolon = V("Semicolon", ";", "separator")
  val DBar = V("DBar", "||", "separator")
  val Bar = V("Bar", "|",  "separator")
  val Question = V("Question", "?","separator")
  val DLt = V("DLt", "<<", "separator")
  val LArrow = V("LArrow", "<-", "separator")
  val Lt = V("Lt", "<", "separator")
  val DGt = V("DGt",  >", "separator")
  val Ge = V("Ge",  =", "separator")
  val Gt = V("Gt",  ", "separator")
  val ColonMinus = V("ColonMinus", ":-", "separator")
  val DColon = V("DColon", "::", "separator")
  val Colon = V("Colon", ":", "separator")
  val Hash = V("Hash", "#", "separator")
  val DPlus = V("DPlus", "++", "separator")
  val Plus = V("Plus", "+", "separator")
  val DMinus = V("DMinus", "--", "separator")
  val RArrow = V("RArrow", "->", "separator")
  val Minus = V("Minus", "-", "separator")
  val Star = V("Star", "*", "separator")
  val Ne = V("Ne", "/=", "separator")
  val Slash = V("Slash", "/", "separator")
  val EEq = V("EEq", "=:=", "separator")
  val ENe = V("ENe", "=/=", "separator")
  val DEq = V("DEq", "==", "separator")
  val Le = V("le", "=<", "separator")
  val Eq = V("Eq", "=", "separator")
  val Exclamation = V("Exclamation", "!", "separator")
}

First, I defined a class V which extends Enumeration.Val, and implemented TokenId with an extra field: fixedText.

Then, I have to explicitly put the value's name to this class and pass it to Enumeration.Val's constructor, so function ErlangTokenId.valueOf(String) will work as Java's enum type.

By type ErlangTokenId = V, type ErlangTokenId.V is now aliased as ErlangTokenId, so you can use ErlangTokenId instead of ErlangTokenId.V everywhere now, which exactly gets the effect of one of the behaviors of Java's enum: enum's value is the same type of enum itself.