Posts for the month of November 2009

How to Setup Dependencies Aware Maven Project for Scala

I have to say, maintain a couple of dependent projects via Ant is a headache, just like me and others may have encountered when use NetBeans' Scala plugin to create and maintain plain Scala projects, these projects, are all Ant based. It's difficult to write a generic Ant template to compute the dependencies graph and then choose the best building path. The building process of cross dependent projects becomes very slow because the redundant enter/exit dependent projects building task.

NetBeans has also the best Maven plugin integrated, I decided to give Maven a trying. This was actually my first attempt to create Maven based project(s). I explored the mini way toward a Scala Maven project, and patched Scalac a bit again to get the dependencies aware compiling working.

Now here's an example I'd like to share with you.

Assume we have n sub-projects, with cross dependencies, for instance, 'lib.util' and 'lib.math', 'lib.indicators' etc. The simplest Maven way is to keep a parent project (I call it 'modules' here) which holds all common settings and module(sub-project) names(paths). The directory structure could be like:

modules
|-- pom.xml
|-- lib.util
|   |-- pom.xml
|   `-- src
|       |-- main
|       |   `-- scala
|       |       `-- lib
|       |           `-- util
|       |               `-- App.scala
|       `-- test
|           `-- scala
|               `-- lib
|                   `-- util
|                       `-- AppTest.scala
|-- lib.math
    |-- pom.xml
    `-- src
        |-- main
        |   `-- scala
        |       `-- lib
        |           `-- math
        |               `-- App.scala
        `-- test
            `-- scala
                `-- lib
                    `-- math
                        `-- AppTest.scala

What I like Maven here is that the parent project 'modules' only maintains the common settings and module paths, all dependencies between sub-projects are then set in sub-project's self pom.xml file. This is a good decoupled strategy in my eyes, you can freely combine these sub-projects at any time without change too much, and each sub-project doesn't need to care about the dependencies of other projects. The parent project is only a centre place for sharing common setting and a centre place to list all available projects and their directories, it's just like a Directory Service.

Now, here is the pom.xml of parent project looks like (all common settings include compiler, repository etc are kept in it):

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>org.aiotrade</groupId>
    <artifactId>modules</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <name>AIOTrade Modules</name>
    <description>Basic Modules for AIOTrade.</description>

    <properties>
        <scala.version>2.8.0-SNAPSHOT</scala.version>
    </properties>

    <repositories>
        <repository>
            <id>scala-tools.release</id>
            <name>Scala-Tools Release Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
        <repository>
            <id>scala-tools.snapshot</id>
            <name>Scala-Tools Snapshot Repository</name>
            <url>http://scala-tools.org/repo-snapshots</url>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.4</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.specs</groupId>
            <artifactId>specs</artifactId>
            <version>1.2.5</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>process-resources</phase> <!-- to support mix java/scala only -->
                        <goals>
                            <goal>add-source</goal>      <!-- to support mix java/scala only -->
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <jvmArgs>
                        <jvmArg>-Xms64m</jvmArg>
                        <jvmArg>-Xmx1024m</jvmArg>
                    </jvmArgs>
                    <scalaVersion>${scala.version}</scalaVersion>
                    <args>
                        <arg>-target:jvm-1.5</arg>
                        <arg>-make:transitivenocp</arg>
                        <arg>-dependencyfile</arg>
                        <arg>${project.build.directory}/.scala_dependencies</arg>
                    </args>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.5</source>
                    <target>1.5</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <reporting>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
        </plugins>
    </reporting>
    
    <modules>
        <module>lib.math</module>
        <module>lib.util</module>
        <module>lib.indicator</module>
    </modules>
</project>

And the pom.xml of sub-project 'lib.util' looks like:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.aiotrade</groupId>
        <artifactId>modules</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <artifactId>lib-util</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>lib-util</name>

</project>

The 'lib.math' one:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.aiotrade</groupId>
        <artifactId>modules</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <artifactId>lib-math</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>lib-math</name>
    
    <dependencies>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>lib.util</artifactId>
            <version>${project.version}</version>
            <type>jar</type>
        </dependency>
    </dependencies>

</project>

That's all the settings, clear and clean.

Now, let's back to another level of dependencies: dependencies between Scala source files. How does this setting get scalac aware dependencies of changed sources?

Again, just like the scalac setting in an Ant based project, where at 'configuration' part of build/plugins/plugin/@maven-scala-plugin, I add "-make:transitivenocp -dependencyfile ${project.build.directory}/.scala_dependencies" to 'args' (should be put in separate 'arg' tag as following)

                <configuration>
                    <jvmArgs>
                        <jvmArg>-Xms64m</jvmArg>
                        <jvmArg>-Xmx1024m</jvmArg>
                    </jvmArgs>
                    <scalaVersion>${scala.version}</scalaVersion>
                    <args>
                        <arg>-target:jvm-1.5</arg>
                        <arg>-make:transitivenocp</arg>
                        <arg>-dependencyfile</arg>
                        <arg>${project.build.directory}/.scala_dependencies</arg>
                    </args>
                </configuration>

Now the last question, how to use NetBeans to create above setting?

Open "File" | "New Project..." | "Maven" | "Maven Project", click "Next", choose "Maven Quickstart Archetype" to create each empty project, then copy/paste above content to corresponding pom.xml, change project groupId, artifactId and version properties. For existed project, copy your source code to "main/scala". You can also mixed Java code, put then at "main/java", NetBeans Scala plugin supports mixed Scala/Java project well.

I'll put the pre-defined archetype on net later, so no need to copy/paste, or, you can create an empty parent/sub-project, commit to your version control system. BTW, I find manually modify pom.xml is a pleasure work.

  • Note No. 1: You have to use/update-to Scala-2.8.0-SNAPHOST with at least scala-compiler-2.8.0-20091128-xxx.jar
  • Note No. 2: If your project is simple, just use NetBeans created Ant based project. If you separate it to a couple of dependent projects, I strongly suggest you to Maven.
  • Note No. 3: A new Scala plugin for NetBeans 6.8RC1 will be available before or at Monday, it's also a must upgrade especially for Windows users.

How to Setup Dependencies Aware Ant Project for Scala

During the past days, I was patching scalac ant task and some relative issues, and now, the dependencies aware ant scalac works (post Scala 2.8.0.r19724).

Below is an example build.xml with dependencies aware setting:

 1 <?xml version="1.0" encoding="UTF-8"?>
 2 <project name="ScalaAntTest" default="build" basedir=".">
 3     <description>Builds, tests, and runs the project ScalaAntTest.</description>
 4 
 5     <property name="src.dir" value="${basedir}/src"/>
 6     <property name="build.dir" value="${basedir}/build"/>
 7     <property name="build.classes.dir" value="${build.dir}/classes"/>
 8 
 9     <target name="init">
10         <property environment="env"/>
11         <condition property="scala.home" value="${env.SCALA_HOME}">
12             <isset property="env.SCALA_HOME"/>
13         </condition>
14         <fail unless="scala.home">set SCALA_HOME first</fail>
15 
16         <property name="scala-library.jar" value="${scala.home}/lib/scala-library.jar"/>
17         <property name="scala-compiler.jar" value="${scala.home}/lib/scala-compiler.jar"/>
18 
19         <path id="build.classpath">
20             <pathelement location="${scala-library.jar}"/>
21             <pathelement location="${scala-compiler.jar}"/>
22             <pathelement location="${build.classes.dir}"/>
23         </path>
24         <taskdef resource="scala/tools/ant/antlib.xml">
25             <classpath>
26                 <pathelement location="${scala-compiler.jar}"/>
27                 <pathelement location="${scala-library.jar}"/>
28             </classpath>
29         </taskdef>
30     </target>
31 
32     <target name="build" depends="init">
33         <mkdir dir="${build.dir}"/>
34         <mkdir dir="${build.classes.dir}"/>
35         <scalac srcdir="${src.dir}"
36                 destdir="${build.classes.dir}"
37                 classpathref="build.classpath"
38                 force="yes"
39                 addparams="-make:transitive -dependencyfile ${build.dir}/.scala_dependencies"
40                 >
41             <src path="${basedir}/src1"/> 
42             <!--include name="compile/**/*.scala"/-->
43             <!--exclude name="forget/**/*.scala"/-->
44         </scalac>
45     </target>
46 
47     <target name="clean" depends="init">
48         <delete dir="${build.dir}"/>
49     </target>
50 
51 </project>

There are some tips here, I'll give a concise explanation:

First, there will be a file call ".scala_dependencies" which is put under "build/" directory after you first clean-build, it will record all dependencies information. Since it's put under "build/", it will be removed automatically after an "ant clean". The "-dependencyfile ${build.dir}/.scala_dependencies" parameter of scalac at line 39 enables this.

Second, you should add "-make:transitive" as scalac's parameter (line 39), which will enable scalac to evaluate the dependencies transitive.

Third, add attribute "force='yes'" (line 38), which tells scalac to check all source files for dependencies and re-compile them if files that dependents on changed.

Forth, you should include "<pathelement location='${build.dir.classes}'>" as part of "build.classpath" (line 22), so scalac won't complain lack of already generated classes when rebuild upon parts of source files.

I've also re-write the project scheme that created by NetBeans plugin, that is, the next released NetBeans Scala plugin will automatically generate dependencies aware build.xml file for new created projects. Unfortunately, you'll have to copy/move your old project src to new created project directory if you want to benefit from it.

For some reasons, "fsc" is no longer supported as the default compiler task in NetBeans created project, which actually brought some annoyances for plugin users. Even without "fsc", the dependencies aware "scalac" should also work satisfiable in most cases.

  • Posted: 2009-11-19 08:00 (Updated: 2009-12-30 04:39)
  • Author: dcaoyuan
  • Categories: Java
  • Comments (0)