AIOTrade, Scala for NetBeans, NLP
For past years, I worked on AIOTrade project, which uses Machine Learning technology to find the structural data pattern.
Then, to find a better programming language for AIOTrade, I tried Erlang, Scala, and wrote IDEs for these candidate languages, I learnt a lot of Formal Language Processing.
Now, finally, I dove into Natural Language Processing, and have a good chance to composite the knowledge of Statistics Learning, Rule based Language Processing on massive structural/un-structural data under parallel computing environment.
The candidate programming language is Scala, I keep to improve the NetBeans support for Scala these days while reading the books/code of NLP. During the long weekend, I've got some improvements on it, here is a summary:
- Migrated to NetBeans' CSL
- Use fsc instead of scalac as the project building compiler
- Fixed setting breakpoint in closure statement
- A basic import-fixer
- Code assistant for local vars and functions
- Run/Debug single file
The plugins can be got for beta testing on NetBeans Developing Update Center (for NetBeans 6.7 beta+), and will be released around the date of official NetBeans 6.7.
Posted at 08:06AM May 06, 2009 by dcaoyuan in AIOTrade | Comments[11]
AIOTrade: Progress of Migrating to Scala - DSL and Parallel Computing
Well, I've migrated the core math, indicator parts of AIOTrade to Scala. There are two noticeable advances for new AIOTrade.
The first is the look and feel of indicator writing. Since Scala is suitable for DSL, now the indicator looks like:
class ARBRIndicator extends ContIndicator {
_sname = "AR/BR"
_grids = Array(50f, 200f)
val period = Factor("Period", 10)
val up = Var[Float]("up")
val dn = Var[Float]("dn")
val bs = Var[Float]("bs")
val ss = Var[Float]("ss")
val ar = Var[Float]("AR", Plot.Line)
val br = Var[Float]("BR", Plot.Line)
def computeCont(begIdx:Int, size:Int) {
for (i <- begIdx until size) {
up(i) = H(i) - O(i)
val up_sum_i = sum(i, up, period)
dn(i) = O(i) - L(i)
val dn_sum_i = sum(i, dn, period)
ar(i) = up_sum_i / dn_sum_i * 100
val bs_tmp = H(i) - C(i)
bs(i) = Math.max(0, bs_tmp)
val bs_sum_i = sum(i, bs, period)
val ss_tmp = C(i) - L(i)
ss(i) = Math.max(0, ss_tmp)
val ss_sum_i = sum(i, ss, period)
br(i) = bs_sum_i / ss_sum_i * 100
}
}
}
Vs Java one:
public class ARBRIndicator extends ContIndicator {
_sname = "AR/BR";
_grids = new Float[] {50f, 200f};
Opt period = new DefaultOpt("Period", 10);
Var up = new DefaultVar("up");
Var dn = new DefaultVar("dn");
Var bs = new DefaultVar("bs");
Var ss = new DefaultVar("ss");
Var ar = new DefaultVar("AR", Plot.Line);
Var br = new DefaultVar("BR", Plot.Line);
void computeCont(int begIdx) {
for (int i = begIdx; i < _itemSize; i++) {
up.set(i, H.get(i) - O.get(i));
float up_sum_i = sum(i, up, period);
dn.set(i, O.get(i) - L.get(i));
float dn_sum_i = sum(i, dn, period);
ar.set(i, up_sum_i / dn_sum_i * 100);
float bs_tmp = H.get(i) - C.get(i);
bs.set(i, Math.max(0, bs_tmp));
float bs_sum_i = sum(i, bs, period);
float ss_tmp = C.get(i) - L.get(i);
ss.set(i, Math.max(0, ss_tmp));
float ss_sum_i = sum(i, ss, period);
br.set(i, bs_sum_i / ss_sum_i * 100);
}
}
}
The apply method from Scala simplifies setter/getter, which makes the formulator looks more natural.
The second is by implementing each indicator as Actor, the computing procedure of indicators can be easily distributed to multiple CPU cores, with so few code modification:
case object Compute
trait Computable extends Actor {
// ----- actor's implementation
def act = loop {
react {
case (Compute, fromTime:Long) => computeFrom(fromTime)
case _ =>
}
}
// ----- end of actor's implementation
def computeFrom(time:Long) :Unit
def computedTime :Long
def factors :ArrayBuffer[Factor]
def factors_=(factors:ArrayBuffer[Factor]) :Unit
def factors_=(values:Array[Number]) :Unit
def dispose :Unit
}
Computable is an Interface/Trait with sync method: computeFrom(Long), now by extending Computable with Actor, implementing a simple function act with react message processing block, all indicators (which extended Computable) can benefit from parallel computing now by calling:
indicator ! (Compute, fromTime)
instead of
indicator.computeFrom(time)
I've done some testing on my 4-core machine, which occupied about 380% CPU usage during running. This is, of course, a most easily implementation for parallel computing under JVM so far.
Another bonus is, I do not need to worry about concurrent calling on computeFrom(Long) now, since all calls will be triggered by 'Compute' messages that are sent to actor's message queue, then be processed sequentially, there is no lock needed any more. The key:
Parallel computing actors + Sequential message driven computing per actor
Posted at 10:07AM Apr 19, 2009 by dcaoyuan in AIOTrade |
AIOTrade Is Migrating to Scala
Finally, after evaluated Erlang, Scala etc, wrote IDE tools for these languages, I began to migrate AIOTrade from Java to Scala, with the help of Scala for NetBeans of course.
The first step is re-writting basic modules to Scala smoothly, with little functional style code; Then, I'll re-design the APIs by Scala's advanced features, including function, trait, actors etc.
Since AIOTrade is a NetBeans suite project, with several NetBeans style modules integrated, I need a general purpose build.xml to get Scala based modules working. Here are build.xml and scala-build.xml, which can be used to write Scala based NetBeans platform modules.
First, you should create a regular NetBeans module project, then put/replace these ant files under your project's base director. You also need to create 2 NetBeans lib wrapper modules, one is for scala-library.jar, another is for scala-compile.jar as parts of your NetBeans suite project (or, check them out from AIOTrade's source repository)
build.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project name="lib.math" default="netbeans" basedir=".">
<import file="scala-build.xml"/>
</project>
scala-build.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project name="scala-module" default="netbeans" basedir=".">
<import file="nbproject/build-impl.xml"/>
<target name="scala-taskdef" depends="init">
<property name="scala.library" value="${cluster}/modules/ext/scala-library.jar"/>
<property name="scala.compiler" value="${cluster}/modules/ext/scala-compiler.jar"/>
<property name="scala.libs" value="${scala.library}:${scala.compiler}"/>
<echo message="cluster: ${cluster}"/>
<echo message="Compiling scala sources via ${scala.library}, ${scala.compiler}"/>
<taskdef resource="scala/tools/ant/antlib.xml">
<classpath>
<pathelement location="${cluster}/modules/ext/scala-library.jar"/>
<pathelement location="${cluster}/modules/ext/scala-compiler.jar"/>
</classpath>
</taskdef>
</target>
<property name="jar-excludes" value="**/*.java,**/*.form,**/package.html,**/doc-files/,**/*.scala"/>
<target name="compile" depends="init,up-to-date,scala-taskdef" unless="is.jar.uptodate">
<!-- javac's classpath should include scala.library and all these paths of "cp" -->
<path id="javac.cp">
<pathelement path="${scala.libs}"/>
<pathelement path="${module.classpath}"/>
<pathelement path="${cp.extra}"/>
</path>
<!-- scalac will check class dependencies deeply, so we can not rely on public package only which is refed by ${module.classpath} -->
<path id="scalac.cp">
<pathelement path="${scala.libs}"/>
<pathelement path="${module.run.classpath}"/>
<pathelement path="${cp.extra}"/>
</path>
<mkdir dir="${build.classes.dir}"/>
<depend srcdir="${src.dir}" destdir="${build.classes.dir}" cache="build/depcache">
<classpath refid="scalac.cp"/>
</depend>
<!-- scalac -->
<scalac srcdir="${src.dir}" destdir="${build.classes.dir}" encoding="UTF-8" target="jvm-${javac.target}">
<classpath refid="scalac.cp"/>
</scalac>
<!-- javac -->
<nb-javac srcdir="${src.dir}" destdir="${build.classes.dir}" debug="${build.compiler.debug}" debuglevel="${build.compiler.debuglevel}" encoding="UTF-8"
deprecation="${build.compiler.deprecation}" optimize="${build.compiler.optimize}" source="${javac.source}" target="${javac.target}" includeantruntime="false">
<classpath refid="javac.cp"/>
<compilerarg line="${javac.compilerargs}"/>
<processorpath refid="processor.cp"/>
</nb-javac>
<!-- Sanity check: -->
<pathconvert pathsep=":" property="class.files.in.src">
<path>
<fileset dir="${src.dir}">
<include name="**/*.class"/>
</fileset>
</path>
</pathconvert>
<fail>
<condition>
<not>
<equals arg1="${class.files.in.src}" arg2=""/>
</not>
</condition>
You have stray *.class files in ${src.dir} which you must remove.
Probably you failed to clean your sources before updating them.
</fail>
<!-- OK, continue: -->
<copy todir="${build.classes.dir}">
<fileset dir="${src.dir}" excludes="${jar-excludes}"/>
</copy>
</target>
<target name="do-test-build" depends="projectized-common.do-test-build">
<scalac srcdir="${test.unit.src.dir}" destdir="${build.test.unit.classes.dir}" excludes="${test.excludes}"
encoding="UTF-8">
<classpath refid="test.unit.cp"/>
</scalac>
</target>
</project>
BTW, the new source code of AIOTrade is controlled under Mercurial version control system on sourceforge.net, you can clone or brows the code at: AIOTrade source repository
Note: The whole project can not be successfully built yet.
Posted at 01:34PM Apr 10, 2009 by dcaoyuan in AIOTrade | Comments[4]
AIOTrade Applet Demo Fixed Timezone Issue
I fixed AIOTrade timezone issue. And, the applet demo on aiotrade.com does not need to be a signed applet any more.
For recent experience, I'm re-considering the RIA platform choice. May applet be saved in the near future? or Flash, Javascript win? First, Java applet needs a whole new look and feel theme, which make it fitting into the web page, second, how can a Java applet access DOM tree easily?
But,
When Flash/Flex tries to take the place of Java applet, it will become another Java itself.
Posted at 12:21PM Mar 27, 2009 by dcaoyuan in AIOTrade | Comments[2]
New Face of aiotrade.com with AIOTrade Applet Demo
I've done some preliminary re-design on aiotrade.com, put the AIOTrade applet on home page, it requires browser with Java plugin above JRE 1.5, the size is about 288k. You can input the symbol to get real-time quote charting. Symbol is same as Yahoo! finance. The quote data is also from Yahoo! finance.
There are still some bugs on it. I'll continue to improve it.
Here's the snapshot of whole home page after you input a symbol. You can remove the chart by press "Remove Quotes" button.
Posted at 06:39PM Mar 14, 2009 by dcaoyuan in AIOTrade | Comments[11]
AIOTrade - It's not Flash or A Picture It's A Java Applet on Web Page
For RIA framework, there are choices of Flex/Flash, Applet/JavaFX etc. I saw a lot of real-time financial charts were written in Flex/Flash, all over the world wide web. Then, how about a Java applet financial charting after Java 6u10? I'd like a try, and here is the result:
- The whole size of a most featured AIOTrade is less than 300k
- On my Macbook, with Java 6u12 installed, the init time is about 1 sec.
- By properly arranging Swing layout, the applet can auto-scale to proper size fit in html page just like any element.
Posted at 03:13PM Mar 11, 2009 by dcaoyuan in AIOTrade | Comments[2]
The Year That Will Be
It's 2009 now, in Beijing.
1==0.999999999......
I met Erlang 2 years ago, which finally brings me to Scala. I learnt a lot from Erlang, and I entered the Scala world with Erlang atmosphere surrounding me. The FP, the Pattern Match, the Actor/Process, I found these familiar friends in Scala everywhere.Scala has extra bonus, to me, static types and OO/FP. The domains I face are usually with a lot of business logic, or, the worlds I try to describe are not only messages, they are, models I don't think are suitable to describe in Function only.
The world itself is OO/FP mixed, like Martin's quote: Two sides of coin. It's something like the Particle/Wave in Quantum. The world is an infinite whole, but the reason of Human Being is always finite, we are using out finite reason to measure the infinite world, it's an unsolvable contradiction: Infinity vs Finite. We have to read our world in OO and, in FP, in snapshot and in continuation.
There won't be "Super Hero" in computer languages, the world is getting self-organization and harmony, so do the languages. Each language is living in an eco-system, born, growing via interacting with environment, disappear ...
The Economy
It was bad in 2008. I tried to do some computing on stock market based on my neural network. What I can say is it will be swing in the next half-year, no big drop, no big rise. The Shanghai Stock Index will swing between 1200 and 3000. At least, no big worse any more.My Self
I need to make some big decisions in this a year.
Posted at 06:40AM Jan 01, 2009 by dcaoyuan in AIOTrade | Comments[2]
Vacation for Snowboarding
I'll take my vacation for snowboarding, 3-4 days. Happy Chinese New Year!
Posted at 08:33PM Feb 07, 2008 by dcaoyuan in AIOTrade | Comments[1]
Scala for NetBeans Screenshot#7: Working on Debug II
So, I've found the cause of that can't add breakpoints on context of object. By setting the enclosing class name of object as object's name + "$", I can now add breakpoints upon object.
To reach here, I have to write an EditorContextImpl for Scala, which will get all compilation information from Scala semantic analyzer. But, to get all debugging features working, I have still to process a bit more conditions. After that, I'll release these modules for public trying.
Click on the picture to enlarge it
Posted at 03:55AM Jan 31, 2008 by dcaoyuan in AIOTrade | Comments[1]
AIOTrade Is Open for Submissions
Here's the announcement from Moshe at the AIOTrade project page on sourceforce.net:
Lately I was added by as a developer in the project and I created a branch called "opencommunity". This branch should contain a development version that is built by contribution of the community in order to answer better the needs of a trader.
Any contribution, any code, designs, ideas, bug fixes, etc., should be send to me and I'll integrate it (the contributers will be credited, of course)
I can be contacted through the forum of the AIOTrade project or directly through aiotrade -d0t- submit -at- gmail -d0t- com or by using the secure client at https://sourceforge.net/sendmessage.php?touser=930864
Moshe
Good works! Moshe
Posted at 03:34PM Jul 06, 2007 by dcaoyuan in AIOTrade | Comments[2]
AIOTrade Goes to BSD License Again
Updated June 21: I've updated the source code in trunk to BSD License.
I decide to re-open AIOTrade, an open-source, free stock technical analysis and charting platform based on NetBeans to BSD license. And welcome the new developer Moshe who will maintain a community branch.
As I'm working hard for my friends project, I have no much time to go with this project in the near future, so, more community members will be helpful for this project.
I'll update the source tree in trunk to BSD license in one week.
I have now a brand new Macbook in white, with 2G memory and Parallels+Windows XP installed, so I can make sure AIOTrade and ErlyBird being Mac OS friendly.
Posted at 06:19PM Jun 21, 2007 by dcaoyuan in AIOTrade | Comments[4]
AIOTrade: Yahoo! data module updated
Due to the recent date format change in the Yahoo! historical data, I've updated the relative module: org-aiotrade-platform-modules-dataserver-basic.nbm
And also updated module: org-aiotrade-math.nbm to fix the bug mentioned at: Solution for bug: wrong date shown
To update your installed Aiotrade 1.0.3:
1.From menu: Tools -> Update Center
2.In the "Select Location of Modules" pane, click "Install Manually Downloaded Modules(.nbm Files)", then "Next"
3.Click [Add...] button, go to the path to select these two .nbm files that you downloaded. (Press Ctrl key down and click on file name to select multiple files)
4.Following the instructions to install updated modules.
5.Restart Aiotrade.
!Notice, you may need to delete all aiotrade log files, which are located at:
For Windows user:
C:\Documents and Settings\yourusername\Application Data\.aiotrade\1.0.3\var\log\
For Mac user:
${HOME}/Library/Application Support/aiotrade/1.0.3/var/log/
This is a quick fix, the final solution will be included in next release.
Posted at 01:42AM Feb 17, 2007 by dcaoyuan in AIOTrade | Comments[5]
Split the Server Part and Client Part? - Next Generation AIOTrade
I'm thinking to re-design the AIOTrade by splitting it to Server part and Client part. The server part may also run as a standalone web application.
But, I'm also busy on another project, so, the prototype and coding work will be postponed.
Posted at 02:00PM Jan 13, 2007 by dcaoyuan in AIOTrade | Comments[6]
The Design of AIOTrade (Part I)
There are no design documents of AIOTrade released so far, it's because I'm still not satisfied the code and architecture, anyway, I'll try to post some draft overall views of the design in my blog.
Below is a digram of main classes [updated: Dec 10, 2006]:
Click on the picture to enlarge it
Posted at 11:47PM Dec 08, 2006 by dcaoyuan in AIOTrade |
Solution for Bug: wrong date shown
There is a bug affects those who's time zone is not UTC:
For each stock, AIOTrade shows the data of the day after. For example, the data of Friday November the 24th is shown in the application at Friday November the 23th.
This bug will be fixed in next release, by far, you can try to add a a property in aiotrade's config file, which is located at:
path.to.installation/etc/aiotrade.conf
change the line:
default_options="-J-Xms24m -J-Xmx128m --fontsize 11"
to:
default_options="-J-Xms24m -J-Xmx128m --fontsize 11 -J-Duser.timezone=UTC"
Sorry about the inconvenience.
Posted at 08:26PM Nov 27, 2006 by dcaoyuan in AIOTrade | Comments[2]
