Collab-o-writer

Why did Java fail as an internet client platform?

2008-03-15T19:06:00.001-07:00

The reason I reflect on it now is that Flex has gained a lot of traction and I am using it now for a project. It's pretty much the same concept as a Java applet but Adobe has done so much better than Sun. First, the client penetration of the Flash Runtime is higher than the Java runtime. Second, if you do need to install it, the entire runtime is 1.5MB. That is less than the Java runtime back in the year 2000!

To test my software on a Mac, I have to buy one?

2008-03-03T15:25:00.000-08:00

huh? I looked everywhere for some way to virtualize a Mac for testing purposes. Nothing. Based on my research it looks difficult to impossible to install Mac OS X on anything but Mac hardware. It's pretty much Russian Roulette if you try it. It will work or it won't. So I ordered a Mac Mini.

I haven't used a Mac since college. They look beautiful and I am looking forward to having one. However, my mind has been trained to view a computer as a sum of upgradeable parts not a single shrink-wrapped device like an iPod. What would happen if Apple became the dominant computer system. They would have a software + hardware monopoly. Very scary.

SQL Server Reporting Services 2008, I wrote that feature!

2008-02-27T12:29:00.001-08:00

The latest CTP for SQL Server 2008 was released today and it contains a feature I wrote from start to finish. You can now render reports to Microsoft Office Word format.

Why would someone want a report in the Word format? The scenario we pictured was a user would want to create a report they could then edit. For example, a specific use case would be a salesperson that generates an invoice using reporting services and then wants to tweak it a little (apply discounts, add some comps, etc.) before passing it on to the customer.

A friend at Microsoft sent me an email telling me that Steve B. made a point to mention it at the launch event. Cool! I'm sure many readers aren't big fans of Microsoft or binary formats but as a developer, I get a rush knowing that millions of people will be using something I created.

I spent most of my time at Microsoft working on a team that completely re-wrote the reporting engine for SQL Server 2008 for performance. I wrote the Word renderer towards the end of my time there after the re-write was tapering off.

Since performance was a central theme of the SSRS 2008 release, the Word renderer was written to be extremely fast and use as little memory as possible. With my casual testing, it would generate large reports in less time and with less memory than Word would use to open the file (I'm sure someone will end up proving me wrong on this :-p).

Anyway, I'm done patting myself on the back. Getting back to work now.

Joel Spolsky's very bad advice

2008-02-26T22:10:00.000-08:00

I was pointed to an essay by good ol' Joel on file format release

It's kind of interesting, he was working for Microsoft in the relatively early days of Office so there are some insights if you're into that sort of thing. I had to laugh when I saw this:

You have a web-based application that needs to output existing Word files in PDF format. Here’s how I would implement that: a few lines of Word VBA code loads a file and saves it as a PDF using the built in PDF exporter in Word 2007. You can call this code directly, even from ASP or ASP.NET code running under IIS. It’ll work. The first time you launch Word it’ll take a few seconds. The second time, Word will be kept in memory by the COM subsystem for a few minutes in case you need it again. It’s fast enough for a reasonable web-based application.

I've done almost exactly what he recommends and it was a mess. One of my responsibilities at an earlier job was to write and maintain a program to automate converting Word documents, along with other file types to PDF in an enterprise server-based application. It seems so simple on the surface but it is a technical mine field.

It was way before Word 2007 (Word 2000 I think) and we used Adobe Acrobat Distiller. This is a "printer" that you can print to and it creates a pdf. There are free ones all over the place now. In Windows, there is a library function called ShellExecute that you can pass in a file name and a verb like "print", "open", etc. and Windows will take care of the rest. This is what we used for most applications.

Distiller had some special features for Word, Excel, and Powerpoint. For example, in Word, it would convert heading levels to bookmarks. These special features were integrated as part of the application and you had to initiate the conversion directly from the application. So this means we had to use VBA to initiate the PDF conversion for Word, Excel, and Powerpoint files.

The first thing we ran into was that the process would appear to hang for no reason, we would go to the server room and there would be a dialog box waiting for someone to click "OK". VBA scripting is very similar to automating a user interface. When Office was launched using VBA, it behaved identically to if a user sitting at the server terminal launched it. We searched for an application or VBA setting to repress any dialogs or other UI interactions. We exhaustively researched this and came to the conclusion that there was no way to prevent it from happening. I had to write an entire scripting language to listen for various window creation events and interact with the UI.

The next problem we ran into was that occassionally, an application would just hang for no reason. It wouldn't bring down the server or anything but it would hold up the conversion queue. We solved this by launching a timer thread that would time out and forcibly kill the process we had launched. This solution led to memory leaks and de-stabilized the server.

Finally, when we would close the application through VBA, sometimes, the process would stick around. This was completely unpredictable. The close function returned normally but if you ran task manager and saw 50 instances of WINWORD.EXE processes you knew sometithing was up.

Last time I talked to my old boss, he told me they had put this on a virtual server. Any time they have problems they just reboot their vm instance. Pretty cool. He did say that my scripting language has grown :-)

The full-text search feature of your typical web app needs work.

2008-02-09T07:26:00.000-08:00

In spite of a couple of dozen alternatives, Google has web-search pretty much locked up at the moment. What I think the world needs is a developer component you could drop into your application and use as the full-text search piece of your software that works as well as Google.

The problem is that any time I use the embedded search feature on a forum, wiki, blog, etc. It reminds me of doing a web search in 1997. It takes a lot of time to narrow down your search terms enough and/or click through pages of results to find what you are looking for. It seems very primitive. Yes, some web-facing apps can be indexed by Google so that is a workaround but many of these sit behind a firewall on intranets.

The Google mini is another solution but it's not something you can ship with your products. So your customers end up having to connect to it and maintain it and this adds to the overall cost of your product. Also, are the relevancy algorithms in the Google mini the same as web search? I would think there should be different heuristics for each app that should be tweakable by the developer.

There is Lucene, but I've been disappointed by the search results in applications that use Lucene for search, so I am assuming that there hasn't been a lot of progress for relevancy algorithms outside of term frequency. Although, it does indexing and searching really well.

This could be an area to focus on for one of the startups who are trying to challenge Google in the web-search market.

Microsoft to make binary file formats documentation public

2008-01-17T10:02:00.000-08:00

I originally heard about this on the POI mailing list and it was later carried on slashdot. To summarize:

Microsoft will let anyone download the full documentation for the binary file formats
Microsoft will start an open source project to create tool to convert from the old binary formats into the new ones.

All of this is supposed to happen on or before Feb 15. I think it's very cool and exciting. It frees me from my NDA. This means I can participate in open source projects that do stuff with the Office binary file formats.

I'm not sure what this means for everyone else. I wouldn't expect APIs to work with the file formats to just start magically appearing. The documentation for the Office 97 file formats is been available since...well 1997. That is what most of my POI work was based on. Also, Microsoft would send anyone a copy of the most recent file format documentation if you sent an email to them and asked for it.

That's what I did when I formed my company. They sent me a hard copy of 5 specs within a week(Word 97-2007, Excel 2007, Excel 97-2003, and Powerpoint 97-2007, Microsoft Drawing format). I was already familiar with these documents because I used them when I worked at Microsoft and at SoftArtisans. They have not improved much since '97. New records have been added for the various new features but they still leave a lot to be desired with notable features being undocumented or underdocumented.

I'm more interested in seeing how this open source project goes. It could be really useful. Haha, I can see them just ripping out the C++ code from Word that does the conversion between doc and docx, slapping a main method on it and dumping it into a subversion repository.

We, the suffering children

2007-11-01T05:41:00.000-07:00

In case you didn't see it already, there is a spat going on between IE and Firefox in the blogosphere. It revolves around javascript.

IE Architect

Firefox Javascript lead

You know what it reminds me of? Two divorced parents who hate each other for the rest of their lives and never stop arguing. Ultimately, it's pointless and the children have to suffer due to the parents' inability to get along. In this case, Microsoft and Firefox are the two parents and we the users are the suffering children :-)

So far, Office 2.0 sucks

2007-10-01T14:44:00.000-07:00

Adobe acquired Virtual Ubiquity. They made a Flex based word processor called Buzzword. I never used it but the screenshots looked pretty slick. There is this sleek black rounded corner look to it. Once you remove the sex appeal what does it bring to the table as far as functionality to the end-user? What is so innovative about it? It's a word processor that runs in a web browser. That seems so 2004.

There is this huge amount of hype surrounding "Office 2.0". I'm getting really tired of it. I want substance. A lot of great ideas out there but the products have not been compelling enough for me to give up my traditional Office programs. They're slow (read Java Swing) and/or they have less features. They work great for writing blog, forum, and wiki posts. When I want to create a professional looking resume or presentation, do I log onto Google Docs? Hell no! I fire up Microsoft Office or OpenOffice. Both apps are lightning fast, they do everything I want and the latter is absolutely free. Then I upload it to Google Docs to share it.

ASP.NET vs. JBoss, Hibernate, Spring, WebWork, Velocity, Sitemesh...

2007-08-22T17:27:00.000-07:00

I saw this blog about Red Hat Developer Studio. There was this comment that struck a chord with me:

Wicket + Spring + Hibernate/JPA + Tomcat, all running in MyEclipse. JSF?
Seam? Ugh. Layers upon layers, and a configuration nightmare.

I love Java. I was working for Microsoft for almost 2 years and it was so refreshing to return to Java development. There is a rich community here and a lot of interesting things happening. That being said, I think our platform can be intimidating to newcomers because we have layers upon layers of abstractions for Web applications. Compare this to the competition, ASP.NET. They have one version of the truth. As a developer on the Microsoft platform, you have to learn a single web application technology, and you're set. Wherever you want to go work, they use the same thing. This leads me to the conclusion that from the standpoint of a business looking to hire developers, the cost for a Window's developers is cheaper because the chances of finding someone who is an expert at ASP.NET are better than the chances of finding an expert at Java web framework X. This worries me.

One thing I thought was odd about ASP.NET is they just seem to ignore the problem of ORM (Object-Relational Mapping). SQL Queries are happily embedded throughout various source files. Many .NET devs just don't see it as that big of a problem I guess. This is probably because Visual Studio provides a GUI for creating and editing SQL and binding data to UI elements.

It's actually really slick for about 75% of what you need to do. That last 25% is so painful. It's difficult to do something in ASP.NET that Microsoft didn't anticipate you doing. For example, in ASP.NET, you can drag and drop a few controls here and there and voila, you have a complete user management system with encrypted passwords and it hooks right into the form authentication scheme for your web app. However, there was one time I wanted to associate my users to a record in a database table in my application. It turned out to be ridiculously difficult since the user database was completely opaque because it was outside of my control.

I've found that Java web frameworks are technically more eloquent solutions. I believe this is because the most popular ones have fought their way to the top based on their merits instead of using a monopoly. But having a unified vision isn't so bad either. I like ASP.NET's simplicity for basic web apps. I wish we had that for Java.

The Word file format as a career

2007-08-01T14:35:00.001-07:00

As I mentioned in an earlier post, I used to work on a technical document management system for NASA. We did a simple analysis and 95% of the hundreds of thousands of documents in our system were binary Word documents (.doc). So I started writing a java library for reading and writing binary Word documents in my spare time to eventually sell as a commercial product. I worked on it on and off and eventually I moved onto other things. Around this time the POI project was formed under Apache Jakarta by Andrew C. Oliver and Marc Johnson. I contacted Andy and donated what I had done so far to POI. This became the codebase for the Word piece of POI (HWPF).

Over the course of 2 years, the project never really took off at POI. I never had the time to really make it into what it could be and I was pretty much the only contributor. It barely reached a semi-working state. I launched a spin-off project so Lucene users would have a simple way to extract text from Word documents here (http://www.textmining.org/)

In 2004, the project I was working on at NASA was waning. I was getting bored so I contacted SoftArtisans and applied for a job. The main selling point was my expertise with the Word file format and SoftArtisans' loved that because one of their products was an API for creating reports in the Word file format. I joined SoftArtisans and became the lead developer for their WordWriter product. This was part of the OfficeWriter suite of products. We released a complete API for reading and writing the Microsoft Word binary file format in late spring 2005. In November of that same year, the OfficeWriter intellectual property was purchased by Microsoft. I moved to Redmond to work on SQL Server Reporting Services as a Microsoft employee.

I left Microsoft in April of this year. What has surprised me over the last few years is that most interest in the Word file format has been for reporting applications. I originally learned the format because I saw the need to extract information from a proprietary format for collaboration. Not the other way around.

A side note about Microsoft. You may be asking yourself why Microsoft would buy an API that reads and writes their own binary format. We were bought by the SQL Server group and my take on it was that the Microsoft Office team is very much against doing anything with the binary formats outside of their respective applications. I'm guessing that from an engineering point of view, the formats (especially Word) have become unmaintainable behomeths. I was told by every person that I came in contact with from the Office team without exception: "Do not, I repeat, do not under any circumstances attempt to write the binary Word file format outside of Microsoft Word, use the new XML format". I chuckle when I see the stories on Slashdot about the file formats (and other MS topics) because it just isn't the way people think it is. They are true believers in the new XML formats ;-)

Ironically, for my own product, I wrote absolutely zero code to work with the Word file format. I'm using a third party java component called Aspose.Words. When I worked at SoftArtisans, they were our main competitor. It has an easy to understand API and most importantly they have excellent support. Most questions I had were already answered in their support forums but when I did post a question, I always received a response from an employee. With all of the hubbub about Office 2.0 and Web 2.0, I'm surprised I haven't seen them mentioned more. They have a full line of Java products that work with Office files on the server. Go check them out.

Introducing the ultimate Wiki editor...Word

2007-08-01T12:50:00.000-07:00

I just released a beta of my product that allows you to use Microsoft Word as a WYSIWYG wiki editor for Confluence. It's called Word/DAV. Go check it out here http://www.benryan.com/.

Personally, I enjoy writing wiki text. It's very much the progammer in me. There is a satisfaction to writing in plain text and then seeing it transformed into something else. 5 years ago, I was working on a technical document management system for NASA. I saw first hand a lot of the pain points of enterprise collaboration. The biggest roadblock I saw to enabling better online collaboration was that most documents were a black box. You couldn't get the content in and out of them without the host application. This made it impossible to do a lot of interesting things that people wanted automated.

Diffing and merging rich content.
Building a document from multiple smaller documents
Email Notifications that include the actual content that changed
Approve/Reject changes in their browser

Around this time I discovered wikis and fell in love. I saw the potential for wikis to be a format-neutral way to store information and to do these types of things. Confluence is by far the best wiki out there and it does some of the above things...but not as well as it could.

I would like to see wikis replace traditional document management systems in the enterprise. Word/DAV is my first step towards that goal. It creates a bridge between the old way of doing things and the new. I hope someone out there finds it useful :-)