Review of Firebird 2.0

Some time ago, many of our users were surprised by the number of features released in Firebird 1.5. Honestly, before preparing this paper, I didn't count new features in Firebird 2.0 and I don't have a clue whether it's superior to the version 1.5 in this aspect or not. But, in my opinion, the major benefit of the version 2.0 is not its feature set.

"What then?", - you ask. I'd describe Firebird 2.0 as "the version which removes the annoying limits". Sounds not so modest, I know. Let me explain better. No doubt that Firebird has a nice multi-generational architecture and a rich SQL language, an embedded usage and a good performance. But I'm sure almost everyone stepped on some internal limitations that worried or even shocked you. To name a few (in no particular order):

Undocumented table limit of about 35GB, an overflow may cause data corruption

Some of them could be of critical priority for your business, others are just annoying. Anyway, I'm glad to report that Firebird 2.0 eliminates most of the aforementioned issues and significantly reduces the impact of others. As for me, this is more important than declaring new language features. But considering also tons of bugfixes and enough new features, Firebird 2.0 definitely represents a major release of your favourite RDBMS. More robust, more feature complete, faster and much more friendly to non-ASCII users.

Of course, there are limits that still exist and many features we don't support yet. But we should have some room for future improvements, shouldn't we? We'll talk about the future development a bit later.

Well, for those who're interested in numbers, let's read the WhatsNew document or Release Notes and make a summary of total changes per version:

Version 1.0: 32 improvements, 55 bugfixes

Version 1.5: 58 improvements, 94 bugfixes

Version 2.0: 82 improvements, 140 bugfixes

Firebird Vulcan

Our primary goal is to merge two codebases (Firebird 2.0 and Vulcan) in order to release Firebird version 3.0. It will be based on the Vulcan tree and will contain its modular architecture and new features, as well as all improvements made in Firebird 2.0. The key features of the Vulcan codebase are:

Globally refactored code

Fine-grained multithreading

Unified provider based architecture

Flexible configuration mechanism

Database level authentication and enhanced security management

Internal DSQL implementation

If you'd ask me to outline the generic development priorities, they would be:

What's lurking in the trees?

As you perhaps know, there are some improvements done by various Firebird developers that didn't go into the 2.0 release due to time constraints. Some of them are included and being tested in Fyracle, others are still in private trees. Also, we still have a few features in Yaffil that requires backporting into Firebird. Everything mentioned above is exactly the new stuff you'll see in version 3.0. Let's see what has been already done:

Firebird 3.0

As both Firebird 2.0 and Firebird Vulcan releases are going to co-exist in the next year, you may ask why version 3.0 is numbered as a major release and what else (except of features already done in both codebases) it will contain. A good question. As we want to shorten the 3.0 release cycle as much as possible, no completely new development is expected to happen in that version. But we need to keep our users interested, so something new should be introduced.

The solution is simple: the 3.0 release is going to incorporate all the work done inside independent branches.

What features to expect?

PRIORITY

shows how much our users want the feature. It's set up after looking at various polls and
forum/newsgroup discussions; also influenced by our analysis of the competitors' feature offerings

COMPLEXITY, based on time/effort estimates made by the core team

NONEAll required work is already done (in some code branch) and just needs to be backported and tested

ImplementationFeature has already been discussed and agreed on, but still requires some minor discussions and actual coding

Design/Implementation

<-Between these two->

DesignWe have basic agreement and some vision of the thing, but the work hasn't been discussed in depth yet, hence we don't have an implementation plan for it

ResearchThe work requires serious analysis before discussing its design and implementation specifications

Admin/Tracing/Monitoring

Asynchronous statement cancellation / timeouts
Force any of the following actions: cancel a running statement, rollback a transaction, kill an attachment. Allow setting timeouts for SQL statements. Should be available at least at the API level.

Detailed SQL tracing/profiling
Show detailed access path (at the RSB tree level) for every retrieval, count rows (profile CPU time, etc) per every node. Available runtime statistics should be extended.

Detailed logging/audit
Allow to log some events happening on the server. These could be: successful/rejected authentification attempts (containing client host info), prepared/executed SQL statements, committed/rolled back transactions, etc. We need API to set up the required events and to retrieve the audit log.

Reliable logical backup
The only case of unrestorable backup should be a physically corrupted backup. Primitive objects (generators, UDFs, etc) must be restored in the beginning. Computed columns and validation constraints must be restored at the end. The engine should reject inconsistent data instead of transform them when reading (e.g. no value -> NULL). GBAK should allow partial restore, driven by switches or interactively.

Point-in-time recovery
The engine must have an optional ability to maintain a redo log in order to roll it over the last logical backup. No data loss is acceptable.

More built-in functions
The SQL-99 (or later) ones (those of major importance for us) must be implemented first. Then we need users feedback about other ones.

Schemas/namespaces
First, it significantly reduces a cost of the issue with short metadata names. Second, it simplifies administration as a number of different databases could be united into a single file. Third, it finally allows us to be fully SQL-92 (entry level) compliant.

Statement/transaction consistency
Solve the known inconsistencies in verbs/transactions. Mostly, this covers blr_for behaviour in INSERT/UPDATE/DELETE statements. Make read-committed transactions compliant with the SQL specification.

Obviously, the list is not complete: it includes only the changes that we consider most important.If this list misses your favourite wish, speak now!

Now let's envisage a matrix where the most preferred and easiest-to-do features are placed in the top-left corner and the harder and/or less wanted ones are toward the bottom-right corner. Moving from one of these corners to another reveals the most likely feature implementation roadmap. Recalling what has been said before, we could imagine a more detailed roadmap:

Firebird 3.0 (the merged version)

Monitoring

Asynchronous statement cancellation

Embedded users / SQL users management

More built-in functions

Temporary tables

SQL functions

Recursive queries

Faster outer joins

SMP support in SS

Compiled statements cache

External functions/procedures

As you can see (and as it has been stated earlier), version 3.0 is expected to include the work already done and a few features that are highly wanted and relatively easy to implement. Everything else tends to slow the development down and hence is excluded from the v.3.0 list.

Firebird 3.0+ (the next major version)

Detailed logging/audit

SQL tracing/profilingUser permissions for metadata

Pluggable authentication modules

Security groups

Long exact numeric implementation

Domains everywhere

Regular expressions

TEXT BLOB compatible with [VAR]CHAR

Reliable logical backup

Optimizer improvements

Statement consistency/atomicy, read committed compliance

Optimized network protocol

Bi-directional indices

Referential integrity without indices

Bulk load/import

It would also be excellent to design schemas/namespaces and longer metadata names for 3.0+, but no promises here. The same for external data sources and deferred constraints.

Of course, some intermediate or minor releases may happen in the meantime, as we'll try to make the release cycles shorter.

This overview is not the last word! A detailed "feature by feature" roadmap for version 3.0 will be published
after the New Year.- D.Y.