Back to the stone age...

Once upon a time there was a computer from IBM. It had 128K. Its process had to page it into 64K chunks. Everyone had to write complicated algorythms, because few things could fit in 64K chunks.

After a while the computer grew and grew until eventually it had gigabytes of ram. People writing programs for it found they could fill it by loading files directly into memory, making their job easier.

Hooray! But then other programs started generating huge files. All of a sudden loading 1 gigabyte of text into memory made the machine crash. But the programmers weren't interested. They just wanted to add coloured text, and tabs, and file explorers to their text editors...

So today's rant is: Why are almost all text editors *stupid* when it comes to reading big files? I want to edit some extracted XML. Its about 1 gig long. I have a bunch of apache logs. Their 1 gig long too. I have 1 gig of memory. Everything seizes up!

Browsing the net for editors, I find hundreds, all with all manner of cool (read pointless) features - one really did have a file exporer in it. All of them use virtual memory, and a huge great textbox/richedit control.

I am actually considering going back to an old dos based editor, as they *couldn't* try and load everything.

Shoddy programming that doesn't deal with boundary cases seems to be everywhere these days. Microsoft are the worst culprit, as its their simplemindedness that leads to complex solutions. Take a databound list. Great! That's saved me lots of time. Then I link it to a table with 10,000 items. Suddenly it doesn't scale so well...

All toolsets suffer from this a bit - they are great with small amounts of data, but as soon as it gets big, you're on your own. The solution seems to be ever bigger amounts of memory, when in fact a well crafted set of controls that support dynamic lazy loading is all thats needed.

Don't get me wrong: I realize that these things are much harder to write. You have to deal with all sorts of sanity problems to avoid these hiccups, but in the core OS wouldn't it be great if you could, say, open a folder with 100,000 files and not have a cup of coffee whilst it opened? Anyone who has ever administereda full Exchange message log will know what I'm talking about.

The sad thing is the OS itself can actually handle such problems well. Its just the interface and control writers that decide to simplify their code rendering the os incapable of handling such things.

It is a *joy* to find an app that can gracefully degrade with volume. MediaWiki (my current plaything) seems have been well designed in this regard. Everything automatically begins to page, and queries are kept short and sweet.

I believe UltraEdit can handle huge files so a big thumbs up to those guys for realizing such things are important.

HP Photo viewing software was also a suprise joy - seemed to sensibly realize that 640 10 Meg Jpgs didn't need to be loaded all at once.

Finally bizzarely old versions of Word were also quite good, but since going fully 32 bit seem to have lost their edge.

The hall of shame is too long to list.

So to any developers out there: remember, not everyone has 10 5K documents on their machine...