Apache Derby Data Corruption

Posted 10 months, 4 weeks ago at 8:44 pm. 0 comments

It saddens me to say, but after a year of fiddling with Apache Derby, working around its quirks, making a custom statement cache so that it’s not so frickin’ slow, that it’s finally time to call it quits and move on to something else.

Data corruption… Over the past year, I have been using Derby as an embedded database inside of Fubario. I’ve been running it on Windows2000, WindowsXP, and OSX 10.4. On all of these OSes, some are laptops, some are desktop machines on all the time, I’ve had varied data corruption scenarios occur, even under relatively little load.

I will admit, I haven’t done a good job keeping records of each kind of data corruption, probably because of foolish optimism that when a big point release of Derby came out it would have solved the problems with the prior version. But alas, every release has had various things go wrong. I’ve had entire databases corrupt so that Derby would refuse to startup, or even an interesting “poison table” corruption where everything was fine until I issue a query against a particular table that was corrupt, at which point Derby went out to lunch.

And when I say little load, I mean pretty much the databases were idling 24×7, with maybe a few thousand rows in them. And to think that I switched to Derby specifically to get more reliability than I had been having with hsqldb or its shiny new cousin h2…

So where am I now? I’ve built a custom flat-file store for Fubario for storing encrypted backup files. It’s pretty simple, really, with a few key characteristics of secret sauce that make it perfect for Fubario and it’s really tiny in terms of amount of code, because it doesn’t have to do very much at all. I’m also using h2 again for storing maybe tens of properties or something like that, miniscule really, probably could use properties in a flat file, but I’m just reusing the sql-driven classes that had been talking to Derby.

I’m done with the 20 second startup and shutdown times.

I’m done with these random and varied and all peculiar data corruption scenarios.

I’m done with foolish optimism? Maybe.