One tool that I hadn’t seen before but have since come to rely on was an innocuous looking button with the label ‘Tidy Array’ that would resize an Excel range with an array formula in it. This is probably the biggest PITA in Excel: entering a formula that returns several values. However, unless you know exactly how many values the formula is going to return, you’re left either getting only part of the result back, or with unsightly #N/As framing the result. Tidy Array would evaluate the formula internally, check the size of the returned result, resize the range in which the formula had been entered, and then re-evaluate the formula to fill that resized range with the result.

Since leaving DKIB, I’ve been playing around with ExcelDNA, Govert van Drimmelen’s fantastic library for building XLLs in managed code, to build an Excel-based interface for my trading system. I missed Tidy Array, so I put something similar together (link below). It’s a little different in that after you run the function (from the ‘Pandora’ menu), it just resizes the selected range to the correct size rather than filling that resized range; the reason for this is that this prevents neighbouring cells from being inadvertently overwritten. (Note: GB’s version had logic to keep it from overwriting stuff, but I prefer to limit function behaviour rather than add behind-the-scenes logic.)

I tried to add a keyboard shortcut for it too (Ctrl + Shift + T) but I haven’t managed to get it to work yet.

To use:

1. Unzip to some convenient directory. Note the location of the TidyArray.xll file.
2. Start Excel and add TidyArray.xll as an Excel Add-in (In 2003, it’s under Tools -> Addins; in 2007, it’s under Excel Options -> Add-ins -> Excel Add-ins

TidyArray.zip

The system is used by a lot of users all over the place for all sorts of things, including, notably, for running risk batches, which of course are crucial for trading. Recently we’d been trying to increase the complexity of batches, so that users could basically tweak all sorts of configuration settings and perform any transformations of market data they may wish to do before running the valuations. There are various ways of doing this, but the most powerful way would be to offer a way to load or inject custom batch logic written in VBA, so that anyone could whip up a custom batch and debug it easily.

In this scenario, the the user would simply specify a text file containing the VBA logic when launching XX in Excel. XX would then run the custom logic and then run the batch. Here’s an example of something we might want to do (‘XX’-prefixed functions are calls to the trading system):

Sub BatchLogic()
    Dim transformations As New Collection
    Call transformations.Add(0.01, "SpotBump")

    Call XXLoadMarketData
    Call XXTransformMarketData(transformations)
End Sub

It sounds a bit hairy given that VBA doesn’t have an evaluate function to execute arbitrary VBA code (you can evaluate worksheet functions, but you don’t have first-class functions like Lisp), but I’m learning you can do practically anything with the VBE if you’re willing to mess around for a bit. The key references, as always, are the Pearson Consulting website and the Erlandsen Consulting website. It took me three tries to get something that worked like I wanted.

My first try was to add a module called ‘Batches’ in the XX VBA project and try to dynimically add a procedure to it and then call it. The code in Batches looked something like this:

Sub InjectAndRunBatchLogic(filename As String)
    Dim text As Collection

    ' Load VBA from file
    text = ReadFileAsCollection(filename)

    ' Add to XX
    Call AddProcedureToModule("Batches", text)

    'Call BatchLogic
    Application.Run ("BatchLogic")
End Sub

The call to BatchLogic is commented out in favour of Application.Run since the function will not exist in XX except when running a custom batch (and will therefore cause compilation problems during normal development).

Unfortunately, it turns out that VBA compiles the module when it is used, so that dynamically modifying the module will either result in BatchLogic not being found (because it didn’t exist when InjectAndRunBatchLogic was called) or Excel crashing (if you try to put a stub BatchLogic in Batches and then delete it before creating it from the text file). Of course, once InjectAndRunBatchLogic has finished running and BatchLogic exists in the module, one can call it. That’s not much use though, because I don’t have the option of restarting the VBA project flow.

The next thing I tried was to dynamically add a new module to XX and then add the new batch procedure to this new module. The idea was that the new module wouldn’t be compiled until BatchLogic was called. Here’s what the code looked like:

Sub InjectAndRunBatchLogic(filename As String)
    Dim text As Collection

    ' Load VBA from file
    text = ReadFileAsCollection(filename)

    ' Add a temporary module to XX
    Call AddModuleToProject("Temporary")

    ' Add to XX
    Call AddProcedureToModule("Temporary", text)

    Application.Run ("BatchLogic")
End Sub

This ended up working decently well, until I password-protected XX’s VBA project so that I could test it the way users use it. Turns out you can’t add modules to password-protected VBA projects. They can’t even add modules to themselves! There’s a workaround using SendKeys that lets you unlock the project, but every reference to the method that I’ve seen warns that it’s flaky at best.

So, back to the drawing board. Since the problem was XX being password-protected, how about creating a new workbook that isn’t password-protected and adding the module there? Worked like a charm. Here’s the code (bit more explicit this time):

Sub InjectAndRunBatchLogic(filename As String)
    Dim wbk As Workbook

    Application.ScreenUpdating = False

    ' Create new workbook
    Set wbk = Workbooks.Add

    With wbk.VBProject
        ' Add module to new workbook (default name: 'Module1')
        .VBComponents.Add (vbext_ct_StdModule)

        ' Add the procedure from the batch file
        .VBComponents("Module1").CodeModule.AddFromFile(filename)

        ' Add reference that points to XX
        .References.AddFromFile(ThisWorkbook.VBProject.filename)
    End With

    ThisWorkbook.Activate
    Application.ScreenUpdating = True

    Application.Run ("'" & wbk.Name & "'!BatchLogic")

    ' Get rid of the new workbook
    wbk.Close (False)
End Sub

You can ignore the ScreenUpdating and workbook activation code — that’s just there to make sure that XX is what remains visible to the user. Since the batch logic will now end up in a separate workbook, you can see I am adding a reference to XX so that it is able to call XX functions. This is actually a fairly nice solution because one can add arbitrary references to the new workbook without having to add them to XX.

So, boys and girls, that’s how that worked out. It’s always fascinating to me how powerful and flexible an environment VBA provides…but of course, that’s also the number one reason for abusing VBA to do things that really should be done in other places.

• WhyNot.net: I was Googling to see if anyone else was curious about using active noise-cancelling techniques for turning down the volume (in some sense) on babies. There are serious technical challenges (such as that current noise-cancelling devices target a point sink), but it’s a cool idea and even if you could get a 50% effective system, it would be immensely useful. And profitable. Anyway, Google turned this up. It’s a bit disappointing that this is another idea that someone’s already thought of, but on the other hand it’s cool to be able to read the comments and see useful extensions (eg, this would be a godsend for snorers’ spouses). The website’s like a catalogue of people’s if-onlys and what-ifs — a goldmine for potential entrepreneurs.

• BookMooch: This one has me seriously excited. I have a bunch of books that I have to lug every time I change flats. I really like most of them but haven’t really picked them up since I first read them. BookMooch is a community of people just like that who are happy to exchange books with each other. Last night, for instance, I got online and listed the 20 or so books I’m willing to give away, and requested a math book that someone in Illinois has. This morning, my inbox had an ‘accepted’ message from the Illinois dude and 4 requests for my books (one from New Zealand!). You might argue sending the book to NZ will pretty much offset whatever money I saved by getting the math book for free, but there’s a larger point here: this is far more efficient than stockpiling books that one never reads. And in return, I get access to a pool of books that might not be available in local bookstores, or even be out of print. Of course, countries like Pakistan have a very active second-hand book market where one can often exchange books with minimal friction/costs so this might prove less useful, but here in the UK, the second-hand book scene is much more limited. There’s an analogy to zakat (and more specifically, keeping money circulating in the economy) taking shape in my head that I think is appropriate; also, it appeases my burgeoning sense of eco- and anti-consumerist responsibility.

• The final Wow will be shared between coLinux and andLinux. coLinux compiled the Linux kernel to run on Windows, where it’s hosted as a guest operating system with virtualised access to your hardware. It’s a really cool idea and its performance is pretty impressive. I haven’t run anything really heavy (ie, OpenOffice), but the things I have run (Perl, Octave, Vim) run superbly well on my Centrino 1.6/1.25GB. The caveat in that last sentence isn’t even that much of a problem for me, since I already have MS Office and am not interested in OO. andLinux built on the coLinux base to provide an easily installable package (2 packages actually: KDE (!) and XFCE). Installation was light and problem-free, and now I can install any additional packages I want with Synaptic. Unlike VMWare, I don’t start the VM manually, and while idle the resource consumption is completely unnoticeable. I have a nice auto-hiding XFCE bar from which I can bring up a terminal or launch the file explorer and that’s that. Linux the way I always wanted it: on my terms, and as I need it.

BTW, my laptop is falling apart. Seriously: it’s cracked in two places, scratched all over, the palmrests are a little worn down, and now the USB and power ports are starting to get loose. It’s lasted 4 years without a single hitch though, which I think is serious testament to how solid Fujitsu machines are.

(The blacked out bit is just the name of my spreadsheet.)

That one is followed immediately by this:

And what, pray, does that mean? Incomplete calculation? OK…except my workbook contains no calculations at all (I’m saving down only values and my pivot table — no formulas), and I get this error when refreshing my pivot table. Invalid field name? Clearly not — I can see the columns are correctly labelled and everything. Naturally, I ran the job by hand and tried refreshing the pivot table to see if I could reproduce the error. Nope. Google? Nope — nothing interesting there.

It turns out the problem is the fact that underlying a pivot table is a pivot cache. This is an Excel object that stores the actual representation of the table in memory and is saved down with your spreadsheet. My jobs would run the calculations, display the results and the pivot table, and then copy them to a fresh workbook in order to save them down. The problem was that I would then close the workbook where the calculations where actually performed: since the pivot table had been created in it, the pivot cache was stored in there, and since it wasn’t saved after the batch was complete (all the results were saved to a new sheet), the pivot cache was lost. So, what Excel was actually trying to say was that it no longer had an internal representation of the pivot table and therefore could not refresh it.

The solution was simply to change the order in which I did things: I now perform my calculations and display the results, and then copy them to a new workbook before producing the pivot table. In this way, the pivot cache is created in the workbook that is to be saved, and there’s no longer a problem.

What converted me, though, was the awesome Firefox extension, which makes del.icio.us bookmarks behave exactly like in-browser bookmarks. It’ll even recommend tags based on your existing bookmarks, which means bookmarking remains an exercise in clicking, rather than forcing you to think and typing in tags. (Eww, thinking…)

So, I’m relatively happy. Except for one little thing: archiving bookmarks. I was a Digg user for a while as well, and while the juvenility (it’s a word; I checked) of the community proved irritating, DuggMirror was one of the highlights. It’s common sense, really: when you have a social news website Digg that routinely sends massive waves of traffic at unsuspecting websites, it’s a good idea to archive the page somewhere (ie, take a snapshot and store it on a different server) so that if the website goes down, you have a mirror to point people to. This is one of the things that Slashdot really missed — sure, Coral/Google/WaybackMachine archives existed, but it was always a bit of a lottery whether they would have the page you wanted, particularly if it was a news page that had just become available.

Anyway, that’s what I want for del.icio.us (they’ve got delicious.com as well if you find the punctuation irritating). I’m a pretty committed bookmarker, and oftentimes (apparently also a word…I blame Amreekans) worry about my bookmarks disappearing. The New York Times, for example, requires a password for articles older than a couple of weeks. There are also transient pages, such as expiring auctions (useful for making comparisons at a later date).

There are bound to be copyright implications, but I doubt it would be a big deal if you only made the pages available to users who bookmarked them (ie, I wouldn’t be able to retrieve pages that you archived); in that case, it would be identical to saving a copy of an article on your hard disk (which I also do…and which has resulted in a 50MB folder of HTML pages I have never accessed again).

Given the sheer number of del.icio.us users, I’m sure there’s some pretty serious Adsense moolah to be made here. Until I get time to do it (or hear of someone else who’s done it already — I honestly don’t care either way, I just want this available), I guess I’ll just have to make do with Errorzilla.