The following appeared in the Firebird Development List from Dimitry Yemanov:

Here I'd like to have the possible additions to the system tables discussed and agreed on. Physical ODS changes will be discussed in a separate thread. As you understand, the sooner we have it resolved, the easier it would be to set up the time frame for the v3.0 Alpha release.

I'll start with my own proposals, expecting others to follow up with both comments/critique and their own suggestions. The changes already done by Adriano are not included here.

Monday, August 24, 2009

Another interesting email conversation between a developer and Dimitry Yemanov (Firebird core developer) produced this information:

Q.What is the intent for the USE_BLOCKING_THREAD? I see it is enabled for all configurations except superserver. What is the rationale for that?

A.The blocking thread (sometimes referred to as the AST delivery thread) listens for the blocking event notification posted by other processes and calls an appropriate BAST (blocking AST) routine inside the local process. This is necessary to asynchronously release or downgrade locks acquired by the current process but requested by other processes and is a vital part of the lock manager.

SuperServer doesn't need this feature as it performs direct AST calls from the lock requester to the lock holder (as everybody lives inside a single process).

The v2.5 embedded engine allows both in-process and cross-process interaction, thus it has the blocking thread enabled. For a single process embedded engine, USE_BLOCKING_THREAD may be undefined.

Friday, August 14, 2009

The following fragment of a message appeared on the Firebird development list. As the question and answer are interesting, I thought I would put it on the blog.

"If I want to make cache flushes to disk very rare, do I set MaxUnflushedWrites & MaxUnflushedWriteTime to -1 or to some large value? What other settings adjustments can make table flushes to disk more rare? Is there any combination of settings that will keep the entire db in cache and never flush it to disk?"

With these settings, Firebird will never explicitly flush the cache to disk. However, flushes may still be performed by the operating system in the background. I believe you understand that these settings are somewhat risky from the database consistency point of view. There is no way to completely avoid flushes at all.

Monday, July 6, 2009

"I'm pleased to announce availability [1] of the first test version ofFB/Java plugin.

FB/Java is a plugin for FB external engines feature, to be presented inv3.0, that allows execution of Java code at the server side. The pluginallows classes/resources in the file system or stored in the database.File system classes/resources are global and the code run with allpermissions. Database classes/resources are per-database and code runsaccordingly with per-engine configured security policy.

The plugin has been made with the assumption that anything could be doneusing existing FB infrastructure. You don't need to setup FTP server orshared directory in the server to update your code or edit security police.

Installation/update of database JARs works from client command lineutility or from SQLJ package procedures. Actually, the same Java classthat run in the client is the one that runs in the server. You can putJARs in the database using a Java URL, so lookup of them is notrestricted to client or server.

Security handling is configured in a central database, with a singlePERMISSION table. Permissions are granted to users that executes theJava code. This way to configure permissions is not adequate for every(or most) cases, and roles are not supported at the moment. I supposeand desire we can have a better design in the future, before a finalrelease.

It's welcome any attempt to bypass the security system. But please,report it. :-)

This version also lacks some notification mechanism for when the codeare going to be unloaded. You may note the problem in the example, rightafter disconnect all attachments from the database the log file remainslocked, as it wasn't been garbaged collected. And no kind of "hotredeployment" support had been implemented or tested. However, I haveideas of how these problems can be solved in very good ways, based onexisting models. For unloading (or any other type of notification),JPA-like listeners. For maintain state after redeployments, serializablesessions.

In the file plugins\java-engine\examples\fbjava\README, you foundinstructions for a "quick start" with it. Inplugins\java-engine\examples\tests there are functional tests for manyof its feature (i.e., how type mappings works, etc).

The plugin is based on Jaybird 2.2 (HEAD). I've put it in a mavenrepository at [2], with its version defined as 2.2-jrt-SNAPSHOT.

The server includes my v3.0 changes, supporting packages, DDL triggers,over clause and some internal changes. It may have others instabilities.

If you have a JAVA_HOME environment variable defined, it should justworks without any other configuration. The quick start assumes you haddefined FIREBIRD, ISC_USER and ISC_PASSWORD. In [1], I've zipped pluginand engine together, but plugin installation consists in just unpackingit in FB directory.

Documentation has room for improvements, and is at [3]. I've done itwith FB docbook tools.

Tuesday, June 30, 2009

I need to post a note on my blog about the IBPhoenix Web site, it seems we are having problems with the T1 line. It looks like there is a problem in the central phone office, where our T1 line originates. It's been a problem now for several days. The issue has been escalated, so hopefully things will improve shortly.

If you see this - then you made a mistake in the build process. Configure.in when you invoked it, assumed that the size of long was 4, and you are attempting to build Firebird for a 32bit environment. As per the prefix.solaris files, you must set the following before running configure:

export CXXFLAGS='-m64'export CFLAGS='-m64'export LDFLAGS='-m64'

Configure will now automtically realise that you want to build in a 64 bit environment and set the size of long to 8.

Wednesday, June 24, 2009

This message appeared on the Firebird Development list, from Jens Zurawski

"I think from time to time, one should really send some positive feedbackto the people who are involved into the development of the firebirdDBMS. Thank you very much for your efforts to make this software this good!

Just before a week or so we eventually upgraded our production serverfrom linux kernel 2.4 to 2.6.x and the firebird engine from 2.0.x to2.1.2. As some of you might remember, long time ago we had some bigtroubles with kernel 2.6.x and firebird, because our system is undervery heavy contention and we stumbled over many freezes, so we werestuck to the 2.4.x kernel. (In average there are 75-150 simultaneousconnections to one database, in peak times up to 300. The database ismore than 20GBs in size and is serving for a pool of approx. 7 differentapplications, starting from web server (php/java) over some internallogistics processings (java) up to the CRM front end (Win32 native), andadditionally there are frequent update tasks with long runningtransactions (about 10-30 minutes) and massive changes to the databaserecords in them).

Now our old database server broke down (hardware problem) and we had tobuy a new one and this time we needed to go for a 2.6.x kernel becausethe new hardware was not supported from the old kernel. To my pleasantsurprise, the actual FB 2.1.2 is working very well with a 2.6 kernel andthe heavy contention we still have on our database. Only thing one hasto do is to dramatically increase the semaphores. In the first place weincreased the LockSemCount to 250 (which was the limit of the defaultkernel setup) but that wasn't enough. After half of the day firebirdcrashed down with a "Fatal lock manager error: semop failed (acquire),errno: 22", accepted no more new connections and also didn't serve theexisting connections any more. Now after we increased the kernel limitand set the LockSemCount up to 1024, everything is working very well. Nomore freezes, no more crashes. (BTW: ironically this hint I was gettingfrom a guy who just has collected some experiences with an Oracleinstallation on Linux ;-)

So, what I want to say: Thanks a lot for all the good work! firebird isreally my most favourite open source project.

Ah, and the new monitoring tables in FB 2.1 are a gift from heaven! NowI'm able to track down problems from applications which are not writtenby me, because now I can simply monitor hanging transactions or longrunning "sub-optimal" queries and send the developer a hint ;) Alsothanks a lot for this feature!"

Wednesday, April 22, 2009

During the QA process for Firebird V2.1.2 we found that there was a problem with creating a database remotely. In effect the inet server would crash.

On further analysis we found that we have a problem where the function free()(standard C dynamic memory deallocator) was being invoked with the incorrect parameter, an address which had not been allocated using malloc().

After setting up the code to make the inet server call abort we got the following stack trace

From this back trace you can see that free() is called from the operator delete[], where we would have expected MemoryPool::deallocate() to be used. For somereason the default operator delete[] is used.

Strange, how did this code arrive at address 0x010440b8?

After an analysis of nm output for fbintl and the linker's map of symbols Alex found that at this address is in fact located the redefined operator delete[], which was compiled in via ld.cpp. However in the assembly version of the file (ld.cpp), it was correctly redefined (i.e. it was calling MemoryPool::deallocate()) operator delete[].

Houston it looks as if we have a buggy toolchain.

But we also need to take into account that AbstractString dtor() and operator delete[] are also WEAK symbols. Weak meaning that such symbols may be present in many object files, and the linker at link-time will take it from the relevant one. But in this case it looks as if it was taken from another file...

But after we compiled into assembler all of the files that are used in the fbintl library we found that they all contained the correctly redefined operator delete[]. As such there is no use of operator delete[], calling std free() function in our code, but anyway we get such use in the resulting binary.

In effect it looks like weak symbols are broken on MacOSX GCC 4.01 (Apple build 5747). And the only way to fix a problem is to avoid them.

We are currently compiling Firebird on PPC 10.4 (Tiger) using a new switch -fno-weak

fno-weak:"Do not use weak symbol support, even if it is provided by the linker. By default, G++ will use weak symbols if they are available. This option exists only for testing, and should not be used by end-users; it will result in inferior code and has no benefits."

Thursday, April 16, 2009

I have been debugging the fb_inet_server trying to find a problem in Firebird 2.1.2 Classic for PowerPC. That takes a little bit of setting up, but I will try and explain that in another blog post. Part of one of the things that I wanted to check was what malloc was doing. However by default any "error" messages get sent to standard out, however you can set the environment variable MallocLogFile to capture messages to a specified file. But setting that in my normal environment is going to work because the fb_inet_server is actually launched by the LaunchDaemon and is then also forked. So we need to set the environment variable in the LaunchDaemon script directly.

The solution:

You need to edit the xml script that "launches" the fb_inet_server. This file can be found in /Library/LaunchDaemons and is called org.firebird.gds.plist

I am writing to express my disappointment with the quality of your 6 seater circular patio set cover.

I purchased this cover from Chinnor Garden Centre, Chinnor, Oxfordshire for £39.99 on 4th November 2008 while on a short trip to the UK from my home in France.

Within a week the first of the supposedly heavy duty ties had broken – followed by the other 3 within the next month. Attempts to re-tie the broken cords were useless (they simply snapped again) and now the metal eyelets have ripped away from the plastic cover, now making it impossible to secure the cover even with heavy duty polypropylene rope which I purchased.

On removing the cover, it is also apparent that large areas of the plastic coating on the inside of the cover have completely worn away and it is no longer waterproof.We do not live in the Outer Hebrides but south of the Loire valley in France, therefore it has most certainly not been exposed to unusually severe weather here!It is clear that this product is not fit for purpose – it is marketed as being top quality PVC backed polyester, secure in windy conditions and giving full winter protection – it is obvious that after just two and a half months use that it is none of these.

As we live in France and very seldom travel to the UK we are unable to return the cover to the place where we purchased it. I would ask you therefore to refund the cost of the cover.

Friday, February 13, 2009

I am a great fan of the Daily Telegraph Cryptic crossword, and have been doing, or attempting to do this crossword, just about every day since I was a student at University.

I was introduced to it by my Grandmother many years ago, when stayed with her in Banbury.

"We" (and I use We, advisedly) used to do the crossword when we had breakfast, usually my Grandmother doing the crossword and explaining how things worked, whilst every now and then I would have a eureka moment. Anyway enough digression.

I found the following web site that specialises in helping people through the Telegraph Cryptic crossword in a way that is much more friendly than normal "specialist" cryptic sites.

Friday, January 30, 2009

The following was posted by Dimitry Yemanov on the Firebird Development list in response to some questions about Memory Usage within Firebird (Classic)

"Need some help understanding how memory allocation is done per process on Linux 1.5.5 CS. It's understandable that there is some amount of working memory needed to support fb process, plus per connection amount, plus cache depending on database page size and buffer count."

Per process (independent) memory usage:- page buffers (40MB in your case)- metadata cache (cannot be predicted/controlled, depends on the schema complexity, especially number of interdependent PSQL objects)- sort buffers (turned off by default on Classic, see SortMemUpperLimit in firebird.conf)- undo log (cannot be predicted/controlled, wastes memory mostly for multiple batch updates in the same transaction)

"What makes me wander is how concurrent user connection count affects memory utilization per process. Somehow the more users we have on the server - the bigger overhead is added per connection. When comparing mem utilization on same database having single connection and 55 for example - utilized memory delta increases by ~33MB. On server having 160 connections it's almost 2-3 times more. Is this lock manager related data? 30MB just for that?"

The lock manager data is shared, but it's mapped into the address space of the every Classic process. I dunno whether the Linux tools include the mmap'ed memory into the counters you check or not. If so, it gets easy to explain: yes, this increase is caused by the lock table. You can draftly calculate the lock manager memory requirements by this formula:

number of connections * number of page buffers * 100 bytes.

The exact size could be found in the output of fb_lock_print (header section) at runtime. With your quite large (for Classic) page cache settings, the numbers should be nearly the same as you experience.

I am glad to introduce new Firebird facility, based on TraceAPI initially developed by Nickolay Samofatov and maintained and given to us by RedSoft (thanks!).

Below is brief overview of what was done. It will be included into HEAD soon, I hope.

Trace session - set of engine-generated events to trace and placement of trace output. Trace session also have its unique ID, optional name, state, flags, creator user and creation timestamp.

List of available trace events is fixed and determined by the Firebird engine. List of events to trace, which data items to trace and placement of trace output is specified by trace configuration when trace session created.

There are two kinds of trace sessions: system audit and user trace. System audit session is started by the engine itself and obtains configuration from the text file. This file name is set via new setting in firebird.conf and by default is "none", i.e. no system audit is configured. There may be only one system audit trace session, obviously. Configuration file contains list of traced events and placement of trace log(s). It is very flexible and allows to log different set of events for different databases into different log files. Example configuration file is attached.

User trace session is managed by user via Services API. There are five new services for this purposes - to start, stop, suspend, resume trace session and to enlist all known trace sessions.

When user application starts trace session it passed optional name and (mandatory) session configuration. Session configuration is a text. Rules and syntax the same as for configuration file mentioned above except of log placement. Engine stores output of user session in set of temporary files and deleted it automatically. After application starts user trace session it must read session's output from service (using isc_service_query). When application decide to stop its trace session it just do detach from service. Also user may manage (suspend\resume\stop) any own trace session. Administrator allowed to manage any trace session.

If user trace session was created by ordinary user it will trace only connections established by this user.

User trace sessions is not preserved when all Firebird processes is stopped. I.e. if user started trace session and Firebird SS or SC process is shutted down, this session is stopped and will not be restarted with new Firebird process. For CS this is not the case as user trace session can't live without connection with service manager and dedicated CS process.

There are three main use cases:

1. Constant audit of engine. This is served by system audit trace.

2. On demand interactive trace of some (or all) activity in some (or all) databases. Application start user trace session and show traced events to user in real time on the screen. User may pause\continue trace and at last stop it.

3. Aggregate of some activity for a relatively long period of time (few hours or even whole day) to analyse it later.Application start user trace session and save trace log in file (or set of files). Session must be stopped manually by the same application or by another one.

Firebird's service manager utility (fbsvcmgr.exe) may be used to manage trace sessions. Later specialized console utility will be introduced (at least i plan to develop it soon after beta). Of course third party vendors are welcome to implement own GUI based monitoring and trace applications.