One tool that I hadn’t seen before but have since come to rely on was an innocuous looking button with the label ‘Tidy Array’ that would resize an Excel range with an array formula in it. This is probably the biggest PITA in Excel: entering a formula that returns several values. However, unless you know exactly how many values the formula is going to return, you’re left either getting only part of the result back, or with unsightly #N/As framing the result. Tidy Array would evaluate the formula internally, check the size of the returned result, resize the range in which the formula had been entered, and then re-evaluate the formula to fill that resized range with the result.

Since leaving DKIB, I’ve been playing around with ExcelDNA, Govert van Drimmelen’s fantastic library for building XLLs in managed code, to build an Excel-based interface for my trading system. I missed Tidy Array, so I put something similar together (link below). It’s a little different in that after you run the function (from the ‘Pandora’ menu), it just resizes the selected range to the correct size rather than filling that resized range; the reason for this is that this prevents neighbouring cells from being inadvertently overwritten. (Note: GB’s version had logic to keep it from overwriting stuff, but I prefer to limit function behaviour rather than add behind-the-scenes logic.)

I tried to add a keyboard shortcut for it too (Ctrl + Shift + T) but I haven’t managed to get it to work yet.

To use:

1. Unzip to some convenient directory. Note the location of the TidyArray.xll file.
2. Start Excel and add TidyArray.xll as an Excel Add-in (In 2003, it’s under Tools -> Addins; in 2007, it’s under Excel Options -> Add-ins -> Excel Add-ins

TidyArray.zip

All I want right now, is for someone to look at Tim Thomas’ line in the March 28 Bulls-Pacers game and restate the value of +/-, basketball’s statistic of the moment:

                        MIN  FGM-A  3PM-A  FTM-A  OREB  DREB  REB  AST  STL  BLK  TO  PF  +/-  PTS
Tim Thomas, PF  6   0-3     0-1        0-0      0   1   1   0   0   0    0    2 +8  0

Yes, for the 6 minutes in which he netted 1 rebound, 3 misses and 2 fouls, he was rewarded with a +8 rating.

Kamen … said every entrepreneurial innovator he’s ever seen shares a few characteristics.

“It’s not that they’re brilliant or well-educated,” Kamen said. “They work all the time. They don’t let failure demoralize or destroy them. They pick themselves up and keep going and eventually, every once in a while, one of your ideas actually breaks through and works, and it makes all that stuff seem worthwhile.”

In other words, Mr. Kamen agrees with Ammi :)

The system is used by a lot of users all over the place for all sorts of things, including, notably, for running risk batches, which of course are crucial for trading. Recently we’d been trying to increase the complexity of batches, so that users could basically tweak all sorts of configuration settings and perform any transformations of market data they may wish to do before running the valuations. There are various ways of doing this, but the most powerful way would be to offer a way to load or inject custom batch logic written in VBA, so that anyone could whip up a custom batch and debug it easily.

In this scenario, the the user would simply specify a text file containing the VBA logic when launching XX in Excel. XX would then run the custom logic and then run the batch. Here’s an example of something we might want to do (‘XX’-prefixed functions are calls to the trading system):

Sub BatchLogic()
    Dim transformations As New Collection
    Call transformations.Add(0.01, "SpotBump")

    Call XXLoadMarketData
    Call XXTransformMarketData(transformations)
End Sub

It sounds a bit hairy given that VBA doesn’t have an evaluate function to execute arbitrary VBA code (you can evaluate worksheet functions, but you don’t have first-class functions like Lisp), but I’m learning you can do practically anything with the VBE if you’re willing to mess around for a bit. The key references, as always, are the Pearson Consulting website and the Erlandsen Consulting website. It took me three tries to get something that worked like I wanted.

My first try was to add a module called ‘Batches’ in the XX VBA project and try to dynimically add a procedure to it and then call it. The code in Batches looked something like this:

Sub InjectAndRunBatchLogic(filename As String)
    Dim text As Collection

    ' Load VBA from file
    text = ReadFileAsCollection(filename)

    ' Add to XX
    Call AddProcedureToModule("Batches", text)

    'Call BatchLogic
    Application.Run ("BatchLogic")
End Sub

The call to BatchLogic is commented out in favour of Application.Run since the function will not exist in XX except when running a custom batch (and will therefore cause compilation problems during normal development).

Unfortunately, it turns out that VBA compiles the module when it is used, so that dynamically modifying the module will either result in BatchLogic not being found (because it didn’t exist when InjectAndRunBatchLogic was called) or Excel crashing (if you try to put a stub BatchLogic in Batches and then delete it before creating it from the text file). Of course, once InjectAndRunBatchLogic has finished running and BatchLogic exists in the module, one can call it. That’s not much use though, because I don’t have the option of restarting the VBA project flow.

The next thing I tried was to dynamically add a new module to XX and then add the new batch procedure to this new module. The idea was that the new module wouldn’t be compiled until BatchLogic was called. Here’s what the code looked like:

Sub InjectAndRunBatchLogic(filename As String)
    Dim text As Collection

    ' Load VBA from file
    text = ReadFileAsCollection(filename)

    ' Add a temporary module to XX
    Call AddModuleToProject("Temporary")

    ' Add to XX
    Call AddProcedureToModule("Temporary", text)

    Application.Run ("BatchLogic")
End Sub

This ended up working decently well, until I password-protected XX’s VBA project so that I could test it the way users use it. Turns out you can’t add modules to password-protected VBA projects. They can’t even add modules to themselves! There’s a workaround using SendKeys that lets you unlock the project, but every reference to the method that I’ve seen warns that it’s flaky at best.

So, back to the drawing board. Since the problem was XX being password-protected, how about creating a new workbook that isn’t password-protected and adding the module there? Worked like a charm. Here’s the code (bit more explicit this time):

Sub InjectAndRunBatchLogic(filename As String)
    Dim wbk As Workbook

    Application.ScreenUpdating = False

    ' Create new workbook
    Set wbk = Workbooks.Add

    With wbk.VBProject
        ' Add module to new workbook (default name: 'Module1')
        .VBComponents.Add (vbext_ct_StdModule)

        ' Add the procedure from the batch file
        .VBComponents("Module1").CodeModule.AddFromFile(filename)

        ' Add reference that points to XX
        .References.AddFromFile(ThisWorkbook.VBProject.filename)
    End With

    ThisWorkbook.Activate
    Application.ScreenUpdating = True

    Application.Run ("'" & wbk.Name & "'!BatchLogic")

    ' Get rid of the new workbook
    wbk.Close (False)
End Sub

You can ignore the ScreenUpdating and workbook activation code — that’s just there to make sure that XX is what remains visible to the user. Since the batch logic will now end up in a separate workbook, you can see I am adding a reference to XX so that it is able to call XX functions. This is actually a fairly nice solution because one can add arbitrary references to the new workbook without having to add them to XX.

So, boys and girls, that’s how that worked out. It’s always fascinating to me how powerful and flexible an environment VBA provides…but of course, that’s also the number one reason for abusing VBA to do things that really should be done in other places.

• WhyNot.net: I was Googling to see if anyone else was curious about using active noise-cancelling techniques for turning down the volume (in some sense) on babies. There are serious technical challenges (such as that current noise-cancelling devices target a point sink), but it’s a cool idea and even if you could get a 50% effective system, it would be immensely useful. And profitable. Anyway, Google turned this up. It’s a bit disappointing that this is another idea that someone’s already thought of, but on the other hand it’s cool to be able to read the comments and see useful extensions (eg, this would be a godsend for snorers’ spouses). The website’s like a catalogue of people’s if-onlys and what-ifs — a goldmine for potential entrepreneurs.

• BookMooch: This one has me seriously excited. I have a bunch of books that I have to lug every time I change flats. I really like most of them but haven’t really picked them up since I first read them. BookMooch is a community of people just like that who are happy to exchange books with each other. Last night, for instance, I got online and listed the 20 or so books I’m willing to give away, and requested a math book that someone in Illinois has. This morning, my inbox had an ‘accepted’ message from the Illinois dude and 4 requests for my books (one from New Zealand!). You might argue sending the book to NZ will pretty much offset whatever money I saved by getting the math book for free, but there’s a larger point here: this is far more efficient than stockpiling books that one never reads. And in return, I get access to a pool of books that might not be available in local bookstores, or even be out of print. Of course, countries like Pakistan have a very active second-hand book market where one can often exchange books with minimal friction/costs so this might prove less useful, but here in the UK, the second-hand book scene is much more limited. There’s an analogy to zakat (and more specifically, keeping money circulating in the economy) taking shape in my head that I think is appropriate; also, it appeases my burgeoning sense of eco- and anti-consumerist responsibility.

• The final Wow will be shared between coLinux and andLinux. coLinux compiled the Linux kernel to run on Windows, where it’s hosted as a guest operating system with virtualised access to your hardware. It’s a really cool idea and its performance is pretty impressive. I haven’t run anything really heavy (ie, OpenOffice), but the things I have run (Perl, Octave, Vim) run superbly well on my Centrino 1.6/1.25GB. The caveat in that last sentence isn’t even that much of a problem for me, since I already have MS Office and am not interested in OO. andLinux built on the coLinux base to provide an easily installable package (2 packages actually: KDE (!) and XFCE). Installation was light and problem-free, and now I can install any additional packages I want with Synaptic. Unlike VMWare, I don’t start the VM manually, and while idle the resource consumption is completely unnoticeable. I have a nice auto-hiding XFCE bar from which I can bring up a terminal or launch the file explorer and that’s that. Linux the way I always wanted it: on my terms, and as I need it.

BTW, my laptop is falling apart. Seriously: it’s cracked in two places, scratched all over, the palmrests are a little worn down, and now the USB and power ports are starting to get loose. It’s lasted 4 years without a single hitch though, which I think is serious testament to how solid Fujitsu machines are.