Reverse engineering the DLTK Python plugin example

Now that you've got Eclipse [link to installing eclipse be], I bet you wanna start kicking some ass, right?

Maybe you've been kicking some ass already, and just wish that you could kick a little more. Specifically, "if that thing would only do what you want it to do".

To that end, we're going to focus on the tools you use to do what you do, and in this case, Eclipse plugins. Yay!

Anyways, I'm enamored of some of the features in some of the editor plugins. Really swell stuff.

I use CFEclipse [link] for editing coldfusion files. It's pretty swell, but I wanted to make it sweller. Better than bee's knees. Or at least have occurrence marking capabilities.

So I took a stab at it, having taken a stab at a couple other plugins, and, as with the other plugins, saw ways to do things different. Not exactly originality, as I got the ideas from other places with similar constraints (like when trying to add key-binding to Subclipse, and looking to the CVS plugin for implementation ideas).

Anyways, a lot of this stuff is about abstraction, oddly enough. Instead of coding 5 specific bits of code, write one general one-- or, true-er; one fragile implementation of something core can become a lot of "work around" coding, sorta.

Anyhow, the less code the better, and CFE looked like there was a good bit of code we could get rid of.

I'm leaning towards getting rid of a lot, like a lot a lot, and relying on other plugins for some stuff we currently handle ourselves, but that's another story.

After looking around, I was liking DLTK for a base IDE type deal, and Mark Drew provided a little starting point-- He'd walked through part of the DLTK IDE how-to walk-thru, and put it in the source repository for people to play with.

Since the code for the example python IDE was available (and even though the real Python DLTK plugin is out now, and I could have used it, or a language really really close to CF for a basis, perhaps) I downloaded it, and sorta strolled around.

Mark Mandel had done some work on an ANTLR coldfusion parser, and the Python plugin used a ANTLR parser, so "I was like, hmmm... I wonder if I can just plug this other parser in?"

And because Eclipse JDT is so awesome, I was able to find out pretty quick. Ha ha! (4hrs+ quick)

What I did was, look for patterns, basically. (Good reason to use something nice and specific when naming stuff-- "PythonExampleParser" was pretty easy to refactor to "CFMLParser", roughly.)

Eclipse would ask me "rename all this crap, dude?" and since it was just the example, I was like, "sure".

Before I did that tho, I'd consolidated the 4 example plugins into two plugins-- a "core" and a "ui". I did that by just dragging the packages from one plugin to the other, and copy/paste the various bits of XML from each plugin into the respective plugin.

The idea was to keep the UI (eclipse presentation type stuff-- font colors or whatnot) apart from the core (I've since gone further, and created a plugin for the CF parser, but for dicking around, I could have just copied them all into one plugin, really-- compartmentalizing stuff was kinda an exercise, sorta.).

Annywho, after I consolidated the 4 plugins to 2, I then refactored all the "PythonExample"s to "CFML"s, which was pretty easy since JDT is so "aware".

Then I fired the plugin up in my usual manner: I'm editing the plugin.xml file for the plugin (any one-- they all fire up the same runtime application) with the Plugin Manifest Editor (right-click, "open with", if you got it but it's not a default-- probably comes with PDE, which you'd have to have by now to do any of this anyways), and on the "overview" screen, there's a "Testing" section, under which there's a nice little "Launch an Eclipse application in Debug mode".

I click on that link, and Eclipse fires up a test instance using all the open plugin projects. That's totally configurable, BTW-- under "Run" (top of screen) there's "Run Configurations" where you can configure the Eclipse Application instance that your code runs in-- use this to disable the loading of any plugins that you don't want running in that test/debug instance.

As the Python parser was still parsing, I was like "cool, now to try shoving the CF parser in"-- I was actually aiming for a quick switch, but it wasn't that pretty. Heh.

So I dropped in the CFMLParser, basically. Of course it didn't have the same kind of AST as the python parser (abstract syntax tree, apparently used to map parsed things to other things), so simply using the python stuff was out-- and the python stuff had all the DLTK stuff, for the outline view and whatnot. Sorta. Hell, look at the code [link], you'll see what I mean. Most of the DLTK stuff's in the parser, actually. Well, grammer file, I guess, technically. It's that grammer file that builds the parser and lexer.

Well now, this is getting long...

Luckily, I think that's pretty much it. This is piss poor, as far as a learning aid, I reckon, but maybe with a little editing or something... eh.

So, DLTK is an dynamic language IDE API, basically. It's got some generic-ish stuff you can use to do things like populate the outline view, create custom searches, and apparently mylyn integration (although I haven't explored that aspect yet), etc..

That generic-ish stuff was in the python parser itself (at least some of it), so to get the CF parser to talk to DLTK I think that's what we've got to do-- define functions components and whatnot, map them to the DLTK equivalents.

It really doesn't matter what happens now... I'm just amazed that things could work that simply. By "work", I mean that when I fired up that test instance, and opened a .cfm file with the "CFML Example Editor" *grin*, I saw output from the parser in the console of my main eclipse instance. "entering cfif, leaving cfif". Did a little more refactoring, put in the more abstract cf parser, that uses dictionary files... "isColdfusion: true" started coming out...

Eh. Maybe I should just do a screencast of the basic deal.

And I mean "doesn't matter" because in a few days, I've sorta kinda picked up just an amazing chunk of knowledge. When I first looked at that CFML.g file, or the python_v3.g parser grammer file... wow. I was like? WTF? Ya know?

Then it turns out it's like sorta, maybe, regular expression chaining or something. (Rough analogy, for sure). There's nice GUIs that make it obvious why X does Y or whatever.

Now I'm off onto another entry tho. Guess I'll cut this out, right about...

Installing Eclipse

So you want to work on open source projects, or maybe do some "work" with a kick ass IDE. Here's how I go about doing it!

First, I download the latest Eclipse version. If we're at, um, around an M4 release, I'll generally opt for that. I like to live on the edge, and really, these days, it's where it's at.

Everything is evolving really fast. I love it, although it probably pisses some people off. Eh.

Anyways: I like the package builds-- what I do most of the time is related to the interweb, so I go with the Java EE build, which comes with most everything I need. Mylyn, the Web whatnots, Data Tools... great stuff!

So, anyways, I download (http://phoenix.eclipse.org/packages/, currently) the Eclipse install I want, and then I fire it up, and add any plugins that don't come with it. For me that's a custom compiled "bleeding edge" version of Subclipse, and CFEclipse. That's it, these days. Oh, and Remote Target Management (for the Remote System Explorer, which does SFTP, FTP, Shells, and other cool things). Used to do QuantumDB, JSEclipse, SFTP...

I point it at a temporary workspace first, while installing plugins, and then I switch it to my "main" workspace (I've recently started using multiple workspaces-- it's easy, and keeps stuff faaaast!).

Here is the command I use to start up eclipse with some memory and aggressive garbage collection (yes, you can tune Eclipse just like CF, for what you're doing):

./eclipse3.4M6/eclipse -data Documents/workspace-cfe -clean -vmargs -Xverify:none -XX:+UseParallelGC -XX:PermSize=64M -XX:MaxNewSize=64M -XX:NewSize=64M -Xmx512m -Xms512m

docs about GC: http://wiki.jboss.org/wiki/TuneVMGarbageCollection

I'll generally have one eclipse directory, with the various versions within, and use a symlink or virtual folder to point at my "default" install, and various scripts for starting up specific workspaces. That line above starts up my CFE specific workspace.

Generally, you'll only start to need multiple workspaces if you're working with upwards of 30 or so /heavy duty/ projects. Up until recently, I probably had like 30 little website projects, 9 honking website applications, and the sources for like 20 open source projects. Mylyn makes stuff manageable, as do working sets.

Things were fine, which is fucking super impressive, you know? Seriously, I'd even leave them all open!

Every once in a while crap goes down tho, and a full workspace rebuild took minutes. Plus, searching /everything/ was a PITA. Just freaking oodles of files. Oodles.

So now I've got stuff broke up better. As you'll see in the next entry, where I talk about how I'm working on a DLTK version of CFE.

Mostly this is because, as I said earlier, I'm a mix-master. I'll check out the sources for 5 or 6 related projects while working on one project, since, to be honest, I'm standing on the shoulders of giants anyways.

See, I don't really know /why/ what I'm doing does what it does-- sure, I'm sorta do, perhaps, and it gets fleshed out as I go, but, seriously, I'm standing on the shoulders of giants, as mentioned previously.

Years of playing with random stuff gives me a certain spidy-sense, I reckon, but there is a lot that is a mystery to me, yet that I take advantage of anyways.

Bah. Talking eclipse install here, right? I'll save that other stuff for "how I guess at what to do" or a similarly titled next post.

So, with the new 3.4 update manager, equinox p2, having bundles of plugins is easy. I used to do the whole: create eclipse plugin extension folders for various things, to keep say, eclipse3.3-only plugins in one folder, and eclipse 3.2 and 3.3 plugins in another, or WTP in one, etc.

I stopped doing that a while ago tho... just got things to such a smooth process, I guess, that it was more trouble than it was worth.

Feel free to look into it tho, if you like to try out various plugins without having to hunt through the default giant eclipse plugin folder (for manual un-install/install, mostly).

If you're running a mac, the command line command I posted above will work. You need to edit (I think) a plist file if you want to be able to just click on the icon and have configuration settings work (the .ini file doesn't work on mac).

Since I'm running a macbook now, I created a ec.app folder, with Contents/MacOS/ folders within, and created a script called ec, which contained the command line stuff, so I can click on the icon... pretty retarded, really, because you can just click on script files too (if you tell Finder to open them with the Terminal app), but hey...

[note to self: see if a one-liner like:] echo thatStatCommand > ec.app/Contents/MacOS/ec chmod +x ec.app/Contents/MacOS/ec [would work]

anywaze... wow... guess that's about it, unless folks think of more eclipse questions, or I think of some more tips... look into mylyn, for sure, peoples... yup... that... is.. all... for... no-ow ow ow ow owww w w.

BlogCFC was created by Raymond Camden. This blog is running version 5.9.3.000. Contact Blog Owner