Thursday, March 05, 2009

State of the DLR for Silverlight

If you’re a frequent reader of this blog, you definitely know what’s going on with the DLR in Silverlight. For those who are new, or just don’t know, hopefully this post will clarify some things.

First off: my previous entries about Silverlight since sdlsdk-0.4.0 dropped:

It would seem the next release will be focused around REPLs, testing, scripting support, and maybe some server-side integration pieces. And you’d be right. Proof is in the project’s public source code repository.

But what about the dynamic language runtime and the languages themselves?

There has not been a new release of sdlsdk for many months, and that’s simply because the above features are not finished, NOT because the DLR and languages have not releases anything for Silverlight since then! The DLR released dlr-0.9 and IronPython released ironpython-2.0.1, both including Silverlight binaries. Also, Harry Pierson put together a nightly build site for the DLR, which includes IronRuby and IronPython, and had Silverlight-specific builds.

Then why the sdlsdk project?

sdlsdk is the main project and release vehicle for Chiron and Microsoft.Scripting.Silverlight, which makes writing applications in DLR-languages possible. It’s also the place for collaborating and experimenting with crazy usages of Silverlight+DLR. The DLR and languages are parts of other projects, so I can not justify bumping the version number of sdlsdk simply because a new version of IronRuby, IronPython, and the DLR are in there. The easiest way to update sdlsdk’s DLR+languages is to grab a daily build of the DLR and drop the binaries into sdlsdk’s /bin folder. The next sdlsdk release might include a script to automate that, as long as new builds don’t break it. =)

This is possible because the DLR, IronRuby, and IronPython teams use a shared continuous-integration server that runs test suites in Silverlight on each checkin, ensuring that they all continue to work really well.

Common misconceptions

During development the state of these projects could be completely different within a few weeks, so misconceptions form very fast. Here are ones which beg for correction …

Chiron, a special HTTP server, performs language compilation on the server and generates a .xap file.

FALSE. While Chiron is a special HTTP server, which generates a XAP file for any given directory, it does not perform language compilation on the server! Chiron is a development-time tool which only sticks the script files in the XAP, and they are run in the browser. When putting your app into production, you tell Chiron to write the XAP file to disk, and then you can put your app on any web-server you want.

Chiron’s existence shows the limitation Silverlight imposes by requiring a XAP file. DLLs can be downloaded by the AppManifest.xaml outside the XAP, so the XAP just serves as a AppManifest.xaml container. I’m hoping to get this fixed by Silverlight 4. =\ It also shows a limitation in not supporting continuations, which would let the user script pause while asynchronously downloading a script file, and then resume the initial script. I’ve been prototyping ways to get around this, but nothing working yet.

Also, if you’re using another web-server in your app (like you have a ASP.NET app with a Silverlight control), you can just set up a file-system-watcher and tell Chiron to regenerate the XAP to disk everytime you edit a file, but it will be served with the other webserver. This is used by silverline, a Rails plugin I wrote to integration Silverlight and IronRuby.

This is why you can’t include JavaScript files from inside other JavaScript files, and all your dependencies need to be pre-defined in the HTML file, because JavaScript can’t download the script files while the JavaScript engine is running.

Common interface implementations were inconsistent between languages, so I can’t create a solution where advanced users can pick and choose a language.

FALSE, in theory. The DLR hosting API (contained in Microsoft.Scripting.dll) provides a single set of APIs that can be used to host many languages. For example, Microsoft.Scripting.Silverlight uses the DLR hosting APIs to run any DLR language in Silverlight, based on what extension the entry-point script has (.py, .rb, etc ...). There are simple hosting examples here.

It is the language’s responsibility to implement certain interfaces so they are host-able by the DLR. IronRuby and IronPython are existing examples of that, and there is documentation in the hosting spec on how this all works, so new languages can follow suit.

However, there are always going to be language-specific scenarios which have different ways of using the hosting API, or the hosting API doesn’t support the feature and you have to use the language’s hosting API. In those cases, it’s best to re-think your scripting design and make the hosting code as simple as possible.

Also, a couple months ago, IronRuby’s hosting support was no where near where it is today. Again, things change fast. If you have any feedback on DLR hosting, please ask around on the DLR Discussions List.

And the state is ...

I hope that made some things clear; the DLR, IronRuby, and IronPython are committed to supporting Silverlight developers. Unfortunately, the reverse is less strong, so if this is important to you please talk about it on the silverlight.net forums. Things might be a bit nicer when the inner ring of the DLR (Microsoft.Scripting.Core.dll) is part of the System.Core in Silverlight, but we’ll see.

9 comments:

stimpy77 said...

Heh. As the author of a rant I posted last night to which you seem to be replying, I ...... thank you for the responses.

I'm still not happy with seeing Chiron or anything server-side in the mix of a Silverlight/DLR solution. For that matter, I'm not happy at all with Silverlight requiring a .xap file in the first place. In fact, I've been turned off by Silverlight ever since I saw the requirement. I had big plans with inline XAML and was going to use runtime-loaded DLR scripts. Plans, tossed.

Glad to see Pything/Ruby/JScript.NET language support, but it's just hard for me to call it DLR if it's bundled in a binary file (.xap), even if Chiron doesn't technically *compile*, it still creates a static binary to get the runtime bundled, and that to me is not DLR.

But what the heck, I'm commenting in the context of a DLR hero who knows DLR inside and out. I'll go back to my longing for Active Scripting days, all by myself. :p

stimpy77 said...

"Also, a couple months ago, IronRuby’s hosting support was no where near where it is today."

Good to know, it was in November or December that I attempted to have a generic DLR runtime module and Ruby didn't implement what it should so I threw all of DLR out because it just wasn't consistent between the languages. If this has changed, I'll give it another shot. :)

Jimmy Schementi said...

Welcome. Part in response to you, part "I'm sure others have been wondering the same thing.

>>> I'm still not happy with seeing Chiron or anything server-side in the mix of a Silverlight/DLR solution.

I agree, but a development-time-only solution makes it a bit nicer. The XAP file is suppose to simulate a read-only filesystem for your app ... since Silverlight is very much a desktop framework hoisted into the web.

>>> For that matter, I'm not happy at all with Silverlight requiring a .xap file in the first place. In fact, I've been turned off by Silverlight ever since I saw the requirement. I had big plans with inline XAML and was going to use runtime-loaded DLR scripts. Plans, tossed.

You can still download script/xaml files on demand, that's no problem. Like my main app could download and load a file using the DLR ... but it would have to be a async download. This async requirement means we can't make things work via require or import. And this time it's not Silverlight's fault ... the it's a browser stack restriction. Granted, I could use browser JS to call XmlHttpRequest synchronously, but it could hang the browser and overall isn't a good strategy. XAP works 100% of the time, while a non-XAP solution is like 50% ... once you try to use an external library it sucks. Again, I'm prototyping some things ... so we'll see.

>>> Glad to see Pything/Ruby/JScript.NET language support, but it's just hard for me to call it DLR if it's bundled in a binary file (.xap), even if Chiron doesn't technically *compile*, it still creates a static binary to get the runtime bundled, and that to me is not DLR.

The XAP is for all your dependencies at startup, to work-around the no-synchronous-downloading. After that, you can definitely load XAML/Script/etc from outside the XAP asynchronously. XAP is just a startup-filesystem solution, since scripting languages (except javascript) are so tied to the idea of a filesystem.

>>> Good to know, it was in November or December that I attempted to have a generic DLR runtime module and Ruby didn't implement what it should so I threw all of DLR out because it just wasn't consistent between the languages. If this has changed, I'll give it another shot. :)

Things are much better now. Good luck, and let me know if you have any other issues, or need it to be easier to get to things outside the XAP.

stimpy77 said...

>>>>> I'm still not happy with seeing Chiron or anything server-side in the mix of a Silverlight/DLR solution.<<<<<<

>>I agree, but a development-time-only solution makes it a bit nicer. The XAP file is suppose to simulate a read-only filesystem for your app ... since Silverlight is very much a desktop framework hoisted into the web. <<

OK wait .. *click*! .. Oooh. So are you saying Chiron wasn't intended to be something that runs at runtime? Because throwing that gear in the mix of a running web architecture just seemed awfully vulnerable. If Chiron is used only once in the deployment process, and not as a runtime server of sorts .. well, I don't understand why even that was necessary (why not just use the file system?), but it's definitely a relief and I feel adequately stupid I hope to your liking.

Alexey Zakharov said...

Hi James,

DLR in Silverlight looks very promising. But I've got to main problems with it:

1. DLR assemblies are not included to Silverlight Core. That is why people should download additional 1 megabyte to start using our application. Are microsoft planning to include DLR in SL 3 Core?

2. There is no integration in Visual Studio. Project IronPython shell wasn't updated since IP first version.

Thanks,
Alexey Zakharov.

Jimmy Schementi said...

@stimpy77 The XAP file is basically a filesystem to a SL app, since it doesn't have access to the local filesystem, just like JavaScript in the browser.

The latest release of AgDLR actually includes a auto-xapper for IIS/Cassini, so you can just put your files on the webserver and it will be served as a XAP whenever the directory containing those files is requested ... makes it that much easier.

Jimmy Schementi said...

@Alexey Zakharov To address your concerns:

>>> DLR assemblies are not included to Silverlight Core

True. In SL4 part of the DLR will be in Silverlight, but the languages will never be, since their open-source projects and release 2-3 times faster than Silverlight.

To help with this, you can serve any DLL from outside the XAP (see chiron.exe.config for documentation), which are only downloaded once into the user's browser cache. SL3 improves upon this by packaging up DLLs in SLVX files for networks which forbid the transmission of DLLs.

A lot of the limitations we have for Dynamic Languages is explained here:

http://blog.jimmy.schementi.com/2008/05/story-of-ruby-and-python-in-silverlight.html

>>> There is no integration in Visual Studio. Project IronPython shell wasn't updated since IP first version.

You can download addons to VS for Python/Ruby colorization, and SL apps for Ruby/Python can just be VS Website projects (see http://blogs.msdn.com/webnext/archive/2008/03/30/silverlight-dynamic-languages-in-visual-studio.aspx), so working in VS is definitely possible. You'll be missing intellisence, but having a REPL in your application (http://www.flickr.com/photos/24458122@N00/3039815620/) can make that a ton easier.

What else is stopping you from using VS?

Miguel Madero said...

Jimmy,

You mentioned Silverline, but I wanted to know wif this project has been updated or is still alive. I've not seen any updates since October 2008 and was wondering what was the state of the project.

It seems like a great idea.

Jimmy Schementi said...

I haven't had time to work on it in a while, but I'm planning to put some time into it for RailsConf this year, so expect a major update in May. It is a really fun project, so I can't wait to update it.