(3.2.10a1) As developed in a long thread starting at
http://mail.zope.org/pipermail/zope/2005-July/160433.html
there appears to be a race bug in the Microsoft Windows socket
implementation, rarely visible in ZEO when multiple processes try to
create an "asyncore trigger" simultaneously. ZEO/zrpc/trigger.py was
replaced by ZODB 3.4's version of that file, to work around this bug when
it occurs.

(3.2.10b1) Collector 1900. In some cases of pickle exceptions raised by
low-level ZEO communication code, callers of marshal.encode() could
attempt to catch an exception that didn't actually exist, leading to an
erroneous AttributeError exception. Thanks to Tres Seaver for the
diagnosis.

(3.2.10b2) Nothing done by tpc_abort() should raise an exception.
However, if something does (an error case), BaseStorage.tpc_abort()
left the commit lock in the acquired state, causing any later attempt
to commit changes hang.

(3.2.10a1) Collector 1843. When a non-integer was passed to a method like
keys() of a Bucket or Set with integer keys, an internal error code
was overlooked, leading to everything from "delayed errors" to segfaults.
Such cases raise TypeError now, as intended.

(3.2.9b1) Doing a subtransaction commit erroneously processed
invalidations, which could lead to an inconsistent view of the database.
For example, let T be the transaction of which the subtransaction commit
was a part. If T read a persistent object O's state before the
subtransaction commit, did not commit new state of its own for O during its
subtransaction commit, and O was modified before the subtransaction commit
by a different transaction, then the subtransaction commit processed an
invalidation for O, and the state T read for O originally was discarded in
T. If T went on to access O again, it saw the newly committed (by a
different transaction) state for O:

The ZopeUndo.Prefix class is actually part of ZODB, for the benefit of
ZEO servers talking with Zope application servers. A bugfix to Prefix
comparison in 3.2.7 had the unfortunate side effect of changing the set of
attributes on a Prefix instance, so that trying to mix a pre-3.2.7 ZEO
client with a 3.2.7 ZEO server, or vice versa, could ship a Prefix
instance from the ZEO client that the ZEO server could not make sense of.
The ZEO server could then disconnect the client due to an AttributeError
when trying to reference a non-existent attribute of Prefix. The
implementation of ZopeUndo.Prefix has been changed to tolerate mixing old
and new in both directions. Thanks to Tres Seaver for the diagnosis.

Appropriate implementations of the storage API's registerDB() and
new_oid() methods were added, delegating to the base storage. This was
needed to support wrapping a ZEO client storage as a DemoStorage base
storage, as some new Zope tests want to do.

new_oid()'s undocumented last= argument was removed. It was used
only for internal recursion, and injured code sanity elsewhere because not
all storages included it in their new_oid()'s signature. Straightening
this out required adding last= everywhere, or removing it everywhere.
Since recursion isn't actually needed, and there was no other use for
last=, removing it everywhere was the obvious choice.

A pid file (containing the process id as a decimal string) is created now
for a ZEO server started via runzeo.py. External programs can
read the pid from this file and derive a "signal name" used in a new
signal-emulation scheme for Windows. This is only necessary on Windows,
but the pid file is created on all platforms that implement
os.getpid(), as long as the pid-filename option is set, or
environment variable INSTANCE_HOME is defined. The pid-filename
option can be set in a ZEO config file, or passed as the new --pid-file
argument to runzeo.py.

If available, runzeo.py now uses Zope's new 'Signal' mechanism for
Windows, to implement clean shutdown and log rotation handlers for Windows.
Note that the Python in use on the ZEO server must also have the Python
Win32 extensions installed for this to be useful.

Silent data loss could occur due to BTree conflict resolution when one
transaction T1 added a new key to a BTree containing at least three buckets,
and a concurrent transaction T2 deleted all keys in the bucket to which the
new key was added. Conflict resolution then created a bucket containing the
newly added key, but the bucket remained isolated, disconnected from the
BTree. In other words, the committed BTree didn't contain the new key added by
T1. Conflict resolution doesn't have enough information to repair this,
so ConflictError is now raised in such cases.

Persistent classes should not define __del__() methods. This has been
documented in the ZODB 3.3 programming guide for over a year, but those
changes didn't get copied over to the 3.2 version of the guide.

Repaired subtle race conditions in establishing ZEO connections, both client-
and server-side. These account for intermittent cases where ZEO failed
to make a connection (or reconnection), accompanied by a log message showing
an error caught in asyncore and having a traceback ending with:

UnpicklingError:invalidloadkey,'Z'.

or:

ZRPCError:badhandshake'(K\x00K\x00U\x0fgetAuthProtocol)t.'

or:

error:(9,'Badfiledescriptor')

or an AttributeError.

These were exacerbated when running the test suite, because of an unintended
busy loop in the test scaffolding, which could starve the thread trying to
make a connection. The ZEO reconnection tests may run much faster now,
depending on platform, and should suffer far fewer (if any) intermittent
"timed out waiting for storage to connect" failures.

The .store() and .restore() methods didn't update the storage's
belief about the largest oid in use when passed an oid larger than the
largest oid the storage already knew about. Because .restore() in
particular is used by copyTransactionsFrom(), and by the first stage
of ZRS recovery, a large database could be created that believed the only
oid in use was oid 0 (the special oid reserved for the root object). In
rare cases, it could go on from there assigning duplicate oids to new
objects, starting over from oid 1 again. This has been repaired. A
new set_max_oid() method was added to the BaseStorage class so
that derived storages can update the largest oid in use in a threadsafe
way.

A FileStorage's index file tried to maintain the index's largest oid as a
separate piece of data, incrementally updated over the storage's lifetime.
This scheme was more complicated than necessary, so was also more brittle
and slower than necessary. It indirectly participated in a rare but
critical bug: when a FileStorage was created via
copyTransactionsFrom(), the "maximum oid" saved in the index file was
always 0. Use that FileStorage, and it could then create "new" oids
starting over at 0 again, despite that those oids were already in use by
old objects in the database. Packing a FileStorage has no reason to
try to update the maximum oid in the index file either, so this kind of
damage could (and did) persist even across packing.

The index file's maximum-oid data is ignored now, but is still written
out so that .index files can be read by older versions of ZODB.
Finding the true maximum oid is done now by exploiting that the main
index is really a kind of BTree (long ago, this wasn't true), and finding
the largest key in a BTree is inexpensive.

A FileStorage's index file could be updated on disk even if the storage
was opened in read-only mode. That bug has been repaired.

This happens if an attempt is made to add distinct objects to the cache
that have the same oid (object identifier). ZODB should never do this,
but it's possible for application code to force such an attempt.

An efficient maxKey() method was implemented for the fsIndex class.
This makes it possible to determine the largest oid in a FileStorage
index efficiently, directly, and reliably, replacing a more delicate scheme
that tried to keep track of this by saving an oid high water mark in the
index file and incrementally updating it.

Collector 1503: excessive logging. It was possible for a ZEO client to
log "waiting for cache verification to finish" messages at a very high
rate, producing gigabytes of such messages in short order.
ClientStorage._wait_sync() was changed to log no more than one
such message per 5 minutes.

Collector 1581: When an attempt to pack a corrupted Data.fs file was
made, it was possible for the pack routine to die with a reference to an
undefined global while it was trying to raise CorruptedError. It
raises CorruptedError, as it always intended, in these cases now.

The deallocation function for ExtensionClass instances wasn't
entirely correct when an instance's __del__ method resurrected the
instance. This could lead to segfaults, but only under a debug build
of Python and ExtensionClass.

The encoding of RPC calls between server and client was being done
with protocol 0 ("text mode") pickles, which could require sending
four times as many bytes as necessary. Protocol 1 pickles are used
now. Thanks to Andreas Jung for the diagnosis and cure.

Collector 1541: small changes to zrpc were backported from ZODB 3.3,
in order to play nice with asyncore changes in Python 2.4.

An explicit transaction abort() wasn't clearing connections' memory of
ReadConflictErrors raised during the transaction, and so new changes to
objects that suffered ReadConflictError still couldn't be committed in
the new transaction either. Thanks to Chris McDonough for the diagnosis
and a focused test case.

New exception ConnectionStateError is raised if an attempt is made to close
a connection while modifications are pending (the connection is involved in
a transaction that hasn't been abort()'ed or commit()'ed). Otherwise,
because Connection objects are reused from a pool, and retain their caches
across closing and (re)opening, opening a Connection could pick up data
from a previous, "abandoned" transaction. In the worst case of this, if a
subtransaction was abandoned, close+reopen could leave the subtransaction
changes in cache, but not reflected anywhere in a storage, and POSKeyErrors
could result. See Collector issue 789 for one nasty instance of this.

There are two flavors of ConnectionStateError that may be raised:

"Cannot close a connection joined to a transaction"
"Cannot close a connection with a pending subtransaction"

The first kind means that changes are pending outside of a subtransaction.
The wording is a bit strange in ZODB 3.2; it's phrased this way so that the
message is the same as in ZODB 3.3, where the wording makes better sense
(3.3 has a different transaction model, where data managers (like
Connection) "join" transactions).

Avoiding the new exception should be easy: if you start a transaction,
finish it (do a commit() or an abort()) before closing any connections it
involves. If you see the new exception in "working" code, that code
was working by accident before.

Transaction.begin() is supposed to imply Transaction.abort(), but if
the only pending changes were in subtransactions, the subtransactions
were not aborted by begin(). This has been repaired.

If ReadConflictError was raised by an attempt to load an object with a
_p_independent() method that returned false, attempting to commit the
transaction failed to (re)raise ReadConflictError for that object. Note
that ZODB intends to prevent committing a transaction in which a
ReadConflictError occurred; this was an obscure case it missed.

Collector #1488 (TemporaryStorage -- going backward in time). This
confusion was really due to that the detail on a ConflictError exception
didn't make sense. It called the current revision "was", and the old
revision "now". The detail is much more informative now. For example,
if the exception said:

ZODB.utils.oid_repr() changed to add a leading "0x", and to strip leading
zeroes. This is used, e.g., in the detail of a POSKeyError exception, to
identify the missing oid. Before, the output was ambiguous. For example,
oid 17 was displayed as 0000000000000011. As a Python integer, that's
octal 9. Or was it meant to be decimal 11? Or was it meant to be hex?
Now it displays as 0x11.

fsrefs.py:

When run with -v, produced tracebacks for objects whose creation was
merely undone. This was confusing. Tracebacks are now produced only
if there's "a real" problem loading an oid.

If the current revision of object O refers to an object P whose
creation has been undone, this is now identified as a distinct case.

Captured and ignored most attempts to stop it via Ctrl+C. Repaired.

Now makes two passes, so that an accurate report can be given of all
invalid references.

analyze.py produced spurious "len of unsized object" messages when
finding a data record for an object uncreation or version abort. These
no longer appear.

The ZEO client cache implementation didn't check for temporary cache
files correctly, sometimes creating named files in the filesystem by
mistake (with names like 1-None-0). This could lead to filesystem
permission problems, and other nasty effects. Thanks to Dieter Maurer
for the diagnosis and the fix.

If the system clock "went backwards" a long time between the times a
FileStorage was closed and reopened, new transaction ids could be
smaller than transaction ids already in the storage, violating a
key invariant. Now transaction ids are guaranteed to be increasing
even when this happens. If time appears to have run backwards at all
when a FileStorage is opened, a new message saying so is logged at
warning level; if time appears to have run backwards at least 30
minutes, the message is logged at panic level (and you should
investigate to find and repair the true cause).

Collector #1379: Undo + FileStorage.restore.
In the presence of multiple-object transactions involving undo or
versions, the FileStorage.restore() method could raise an inappropiate
exception (bad argument to file.seek()), or silently fail to find
the data it was looking for. Base Zope makes no use of restore(), but
this method is crucial to ZRS recovery, so this is a critical bugfix
at least for sites running ZRS. Thanks to Tres Seaver and Jim Fulton
for the diagnosis and cure.

As announced with 3.2.2b1, the Berkeley-based storages are no longer
supported. The code for them still exists, but the Berkeley tests have
been disabled (primarily to prevent segfaults on Linux systems with
newer versions of Sleepycat Berkeley DB).

Collector #1208: Infinite loop in cPickleCache.
If a persistent object had a __del__ method (probably not a good idea
regardless, but we don't prevent it) that referenced an attribute of
self, the code to deactivate objects in the cache could get into an
infinite loop: ghostifying the object could lead to calling its __del__
method, the latter would load the object into cache again to
satsify the attribute reference, the cache would again decide that
the object should be ghostified, and so on. The infinite loop no longer
occurs, but note that objects of this kind still aren't sensible (they're
effectively immortal). Thanks to Toby Dickenson for suggesting a nice
cure.

Under exceedingly unlikely conditions, it was possible for DB.open()
to block with as few as two threads active. This was revealed by the
intermittent failure of the checkConcurrentUpdates1Storage test on a
hyper-threaded 3.2GHz P4 machine. That test has never been reported
to fail elsewhere, and didn't fail on that box either if hyper-threading
was disabled in the BIOS. It was a legitimate thread race, but without
hyper-threading it was extremely unlikely to happen.

Doc/storages.html changed to record that the experimental BDB-based
storages are unsupported and will be dropped in ZODB 3.3. This is due to
lack of visible interest, and of resources (whether internal or external)
for support of this code.

When trying to do recovery to a time earlier than that of the most recent
full backup, repozo.py failed to find the appropriate files, erroneously
claiming "No files in repository before <specified time>". This has
been repaired.

Collector #1330: repozo.py -R can create corrupt .fs.
When looking for the backup files needed to recreate a Data.fs file,
repozo could (unintentionally) include its meta .dat files in the list,
or random files of any kind created by the user in the backup directory.
These would then get copied verbatim into the reconstructed file, filling
parts with junk. Repaired by filtering the file list to include only
files with the data extensions repozo.py creates (.fs, .fsz, .deltafs,
and .deltafsz). Thanks to James Henderson for the diagnosis.

Collector #553: sporadic Zope crashes
In rare circumstances, it was possible for BTree conflict resolution
to be handed three empty BTrees (code needed to fetch an empty BTree
from the database, add keys to it, then remove them all again, before
committing the transaction). This could cause a memory fault in C code.

Fixed a serious bug in the new pack implementation. If pack was
called on the storage and passed a time earlier than a previous pack
time, data could be lost. In other words, if there are any two pack
calls, where the time argument passed to the second call was earlier
than the first call, data loss could occur. The bug was fixed by
causing the second call to raise a StorageError before performing any
work.

Fixed a bug in the ZEO authentication implementation that could cause
a properly authenticated connection to fail unexpectedly with a
ValueError -- "Received message without HMAC."

Fixed a bug in the ZEO server's getInvalidations() method. If it had
the requested invalidations, it would send all the invalidations in
its queue rather than just the requested ones. This didn't affect
correctness of the cache, but did cause it to throw out valid data.

Fixed a bug that prevented the -m / --monitor argument from working.

Fixed and enhanced the zeopasswd.py script. It now runs as a script
and supports command-line arguments for all the options that can be
specified in the configuration file.

This code in this release should be identical to the code in Zope
2.7.0 beta 3.

Changed the ZEO server and control process to work with a single
configuration file; this is now the default way to configure these
processes. (It's still possible to use separate configuration files.)
The ZEO configuration file can now include a "runner" section used by
the control process and ignored by the ZEO server process itself. If
present, the control process can use the same configuration file.

Fixed a performance problem in the logging code for the ZEO protocol.
The logging code could call repr() on arbitrarily long lists, even
though it only logged the first 60 bytes; worse, it calls repr() even
if logging is currently disabled. Fixed to call repr() on individual
elements until the limit is reached.

ReadConflictErrors have an experimental new method, ignore(). If a
database connection raises a read conflict, the current transaction
was eventually be aborted. If the ignore() method is called, it will
allow a transaction to commit after receiving a read conflict. This
method should be used with great care, because ignoring read conflicts
may allow inconsistent data to be written to the database.

Several bare string exceptions were changed to raise exception
objects. All these exceptions were in fairly obscure parts of the
code.

Added a summary to the Doc directory. There are several new documents
in the 3.2 release, including "Using zdctl and zdrun to manage server
processes" and "Running a ZEO Server HOWTO."

Fixed ZEO's protocol negotiation mechanism so that a client ZODB 3.1
can talk to a ZODB 3.2 server.

Fixed a memory leak in the ZEO server. The server was leaking a few
KB of memory per connection.

Fixed a memory leak in the ZODB object cache (cPickleCache). The
cache did not release two references to its Connection, causing a
large cycle of objects to leak when a database was closed.

Fixed a bug in the ZEO code that caused it to leak socket objects on
Windows. Specifically, fix the trigger mechanism so that both sockets
created for a trigger are closed.

Fixed a bug in the ZEO storage server that caused it to leave temp
files behind. The CommitLog class contains a temp file, but it was
not closing the file.

Changed the order of setuid() and setgid() calls in zdrun, so that
setgid() is called first.

Added a timeout to the ZEO test suite that prevents hangs. The test
suite creates ZEO servers with randomly assigned ports. If the port
happens to be in use, the test suite would hang because the ZEO client
would never stop trying to connect. The fix will cause the test to
fail after a minute, but should prevent the test runner from hanging.

The logging package was updated to include the latest version of the
logging package from Python CVS. Note that this package is only
installed for Python 2.2. In later versions of Python, it is
available in the Python standard library.

The ZEO1 directory was removed from the source distribution. ZEO1 is
not supported, and we never intended to include it in the release.

Note: The changes listed for this release include changes also made in
ZODB 3.1.x releases and ported to the 3.2 release.

This version of ZODB 3.2 is not compatible with Python 2.1. Early
versions were explicitly designed to be compatible with Zope 2.6.
That plan has been dropped, because Zope 2.7 is already in beta
release.

Several of the classes in ZEO and ZODB now inherit from object, making
them new-style classes. The primary motivation for the change was to
make it easier to debug memory leaks. We don't expect any behavior to
change as a result.

A new feature to allow removal of connection pools for versions was
ported from Zope 2.6. This feature is needed by Zope to avoid denial
of service attacks that allow a client to create an arbitrary number
of version pools.

Fixed several critical ZEO bugs.

If several client transactions were blocked waiting for the storage
and one of the blocked clients disconnected, the server would
attempt to restart one of the other waiting clients. Since the
disconnected client did not have the storage lock, this could lead
to deadlock. It could also cause the assertion "self._client is
None" to fail.

If a storage server fails or times out between the vote and the
finish, the ZEO cache could get populated with objects that didn't
make it to the storage server.

If a client loses its connection to the server near the end of a
transaction, it is now guaranteed to get a ClientDisconnected error
even if it reconnects before the transaction finishes. This is
necessary because the server will always abort the transaction.
In some cases, the client would never see an error for the aborted
transaction.

In tpc_finish(), reordered the calls so that the server's tpc_finish()
is called (and must succeed) before we update the ZEO client cache.

The storage name is now prepended to the sort key, to ensure a
unique global sort order if storages are named uniquely. This
can prevent deadlock in some unusual cases.

Fixed several serious flaws in the implementation of the ZEO
authentication protocol.

The smac layer would accept a message without a MAC even after the
session key was established.

The client never initialized its session key, so it never checked
incoming messages or created MACs for outgoing messags.

The smac layer used a single HMAC instance for sending and receiving
messages. This approach could only work if client and server were
guaranteed to process all messages in the same total order, which
could only happen in simple scenarios like unit tests.

Fixed a bug in ExtensionClass when comparing ExtensionClass instances.
The code could raise RuntimeWarning under Python 2.3, and produce
incorrect results on 64-bit platforms.

Fixed bug in BDBStorage that cause lead to DBRunRecoveryErrors when a
transaction was aborted after performing operations like commit
version or undo that create new references to existing pickles.

Fixed a bug in Connection.py that caused it to fail with an
AttributeError if close() was called after the database was closed.

The test suite leaves fewer log files behind, although it still leaves
a lot of junk. The test.py script puts each tests temp files in a
separate directory, so it is easier to see which tests are causing
problems. Unfortunately, it is still to tedious to figure out why the
identified tests are leaving files behind.

This release contains the latest and greatest version of the
BDBStorage. This storage has still not seen testing in a production
environment, but it represents the current best design and most recent
code culled from various branches where development has occurred.

The Tools directory contains a number of small improvements, a few new
tools, and README.txt that catalogs the tools. Many of the tools are
installed by setup.py; those scripts will now have a #! line set
automatically on Unix.

Fixed bugs in Tools/repozo.py, including a timing-dependent one that
could cause the following invocation of repozo to do a full backup when
an incremental backup would have sufficed.

A pair of new scripts from Jim Fulton can be used to synthesize
workloads and measure ZEO performance: see zodbload.py and
zeoserverlog.py in the Tools directory. Note that these require
Zope.

Tools/checkbtrees.py was strengthened in two ways:

In addition to running the _check() method on each BTree B found,
BTrees.check.check(B) is also run. The check() function was written
after checkbtrees.py, and identifies kinds of damage B._check()
cannot find.

Cycles in the object graph no longer lead to unbounded output.
Note that preventing this requires remembering the oid of each
persistent object found, which increases the memory needed by the
script.

Fixed critical race conditions in ZEO's cache consistency code that
could cause invalidations to be lost or stale data to be written to
the cache. These bugs can lead to data loss or data corruption.
These bugs are relatively unlikely to be provoked in sites with few
conflicts, but the possibility of failure existed any time an object
was loaded and stored concurrently.

Fixed a bug in conflict resolution that failed to ghostify an object
if it was involved in a conflict. (This code may be redundant, but it
has been fixed regardless.)

The ZEO server was fixed so that it does not perform any I/O until all
of a transactions' invalidations are queued. If it performs I/O in the
middle of sending invalidations, it would be possible to overlap a
load from a client with the invalidation being sent to it.

The ZEO cache now handles invalidations atomically. This is the same
sort of bug that is described in the 3.1.2b1 section below, but it
affects the ZEO cache.

Fixed several serious bugs in fsrecover that caused it to fail
catastrophically in certain cases because it thought it had found a
checkpoint (status "c") record when it was in the middle of the file.

Two new features snuck into this beta release.

The ZODB.transact module provides a helper function that converts a
regular function or method into a transactional one.

The ZEO client cache now supports Adaptable Persistence (APE). The
cache used to expect that all OIDs were eight bytes long.

Invalidations are now processed atomically. Each transaction will see
all the changes caused by an earlier transaction or none of them.
Before this patch, it was possible for a transaction to see invalid
data because it saw only a subset of the invalidations. This is the
most likely cause of reported BTrees corruption, where keys were
stored in the wrong bucket. When a BTree bucket splits, the bucket
and the bucket's parent are both modified. If a transaction sees the
invalidation for the bucket but not the parent, the BTree in memory
will be internally inconsistent and keys can be put in the wrong
bucket. The atomic invalidation fix prevents this problem.

A number of minor reference count fixes in the object cache were
fixed. That's the cPickleCache.c file.

It was possible for a transaction that failed in tpc_finish() to lose
the traceback that caused the failure. The transaction code was fixed
to report the original error as well as any errors that occur while
trying to recover from the original error.

The "other" argument to copyTransactionsFrom() only needs to have an
.iterator() method. For convenience, change FileStorage's and
BDBFullStorage's iterator to have this method, which just returns
self.

Mount points are now visible from mounted objects.

Fixed memory leak involving database connections and caches. When a
connection or database was closed, the cache and database leaked,
because of a circular reference involving the cache. Fixed the cache
to explicitly clear out its contents when its connection is closed.

The ZODB cache has fewer methods. It used to expose methods that
could mutate the dictionary, which allowed users to violate internal
invariants.

There is a new HOWTO for running a ZEO server. The draft in this
release is incomplete, but provides more guidance than previous
releases. See the file Doc/ZEO/howto.txt.

The ZEO storage server's transaction timeout feature was refactored
and made slightly more rebust.

A new ZEO utility script, ZEO/mkzeoinst.py, was added. This creates a
standard directory structure and writes a configuration file with
mostly default values, and a bootstrap script that can be used to
manage and monitor the server using zdctl.py (see below).

Much work was done to improve zdaemon's zdctl.py and zdrun.py scripts.
(In the alpha 1 release, zdrun.py was called zdaemon.py, but
installing it in <prefix>/bin caused much breakage due to the name
conflict with the zdaemon package.) Together with the new
mkzeoinst.py script, this makes controlling a ZEO server a breeze.

A ZEO client will not read from its cache during cache verification.
This fix was necessary to prevent the client from reading inconsistent
data.

The isReadOnly() method of a ZEO client was fixed to return the false
when the client is connected to a read-only fallback server.

The sync() method of ClientStorage and the pending() method of a zrpc
connection now do both input and output.

The short_repr() function used to generate log messages was fixed so
that it does not blow up creating a repr of very long tuples.

Trying to store an object of a non-integer type into an IIBTree
or OIBTree could leave the bucket in a variety of insane states. For
example, trying

b[obj] = "I'm a string, not an integer"

where b is an OIBTree. This manifested as a refcount leak in the test
suite, but could have been much worse (most likely in real life is that
a seemingly arbitrary existing key would "go missing").

When deleting the first child of a BTree node with more than one
child, a reference to the second child leaked. This could cause
the entire bucket chain to leak (not be collected as garbage
despite not being referenced anymore).

Most of the changes in this release are performance and stability
improvements to ZEO. A major packaging change is that there won't be
a separate ZEO release. The new ZConfig is a noteworthy addtion (see
below).

An experimental new transaction API was added. The Connection class
has a new method, setLocalTransaction(). ZODB applications can call
this method to bind transactions to connections rather than threads.
This is especially useful for GUI applications, which often have only
one thread but multiple independent activities within that thread
(generally one per window). Thanks to Christian Reis for championing
this feature.

Applications that take advantage of this feature should not use the
get_transaction() function. Until now, ZODB itself sometimes assumed
get_transaction() was the only way to get the transaction. Minor
corrections have been added. The ZODB test suite, on the other hand,
can continue to use get_transaction(), since it is free to assume that
transactions are bound to threads.

There is a new recommended script for starting a storage server. We
recommend using ZEO/runzeo.py instead of ZEO/start.py. The start.py
script is still available in this release, but it will no longer be
maintained and will eventually be removed.

There is a new zdaemon implementation. This version is a separate
script that runs an arbitrary daemon. To run the ZEO server as a
daemon, you would run "zdrun.py runzeo.py". There is also a simple
shell, zdctl.py, that can be used to manage a daemon. Try
"zdctl.py -p runzeo.py".

There is a new version of the ZEO protocol in this release and a first
stab at protocol negotiation. (It's a first stab because the protocol
checking supporting in ZODB 3.1 was too primitive to support anything
better.) A ZODB 3.2 ZEO client can talk to an old server, but a ZODB
3.2 server can't talk to an old client. It's safe to upgrade all the
clients first and upgrade the server last. The ZEO client cache
format changed, so you'll need to delete persistent caches before
restarting clients.

The ZEO cache verification protocol was revised to require many fewer
messages in cases where a client or server restarts quickly.

The performance of full cache verification has improved dramatically.
XXX Get measurements from Jim -- somewhere in 2x-5x recall. The
implementation was fixed to use the very-fast getSerial() method on
the storage instead of the comparatively slow load().

The ZEO server has an optional timeout feature that will abort a
connection that does not commit within a certain amount of time. The
timeout works by closing the socket the client is using, causing both
client and server to abort the transaction and continue. This is a
drastic step, but can be useful to prevent a hung client or other bug
from blocking a server indefinitely.

A bug was fixed in the ZEO protocol that allowed clients to read stale
cache data while cache verification was being performed. The fixed
version prevents the client from using the storage until after
verification completes.

The ZEO server has an experimental monitoring interface that reports
usage statistics for the storage server including number of connected
clients and number of transactions active and committed. It can be
enabled by passing the -m flag to runsvr.py.

The ZEO ClientStorage no longer supports the environment variables
CLIENT_HOME, INSTANCE_HOME, or ZEO_CLIENT.

The ZEO1 package is still included with this release, but there is no
longer an option to install it.

The BTrees package now has a check module that inspects a BTree to
check internal invariants. Bugs in older versions of the code code
leave a BTree in an inconsistent state. Calling BTrees.check.check()
on a BTree object should verify its consistency. (See the NEWS
section for 3.1 beta 1 below to for the old BTrees bugs.)

Fixed a rare conflict resolution problem in the BTrees that could
cause an segfault when the conflict resolution resulted in any
empty bucket.

The distutils setup now installs several Python scripts. The
runzeo.py and zdrun.py scripts mentioned above and several fsXXX.py
scripts from the Tools directory.

The test.py script does not run all the ZEO tests by default, because
the ZEO tests take a long time to run. Use --all to run all the
tests. Otherwise a subset of the tests, mostly using MappingStorage,
are run.

There are two new storages based on Sleepycat's BerkeleyDB in the
BDBStorage package. Barry will have to write more here, because I
don't know how different they are from the old bsddb3Storage
storages. See Doc/BDBStorage.txt for more information.

It now takes less time to open an existing FileStorage. The
FileStorage uses a BTree-based index that is faster to pickle and
unpickle. It also saves the index periodically so that subsequent
opens will go fast even if the storage was not closed cleanly.

The new ZConfig package, which will be used by Zope and ZODB, is
included. ZConfig provides a configuration syntax, similar to
Apache's syntax. The package can be used to configure the ZEO server
and ZODB databases. See the module ZODB.config for functions to open
the database from configuration. See ZConfig/doc for more info.

The zLOG package now uses the logging package by Vinay Sajip, which
will be included in Python 2.3.

The Sync extension was removed from ExtensionClass, because it was not
used by ZODB.

A new feature to allow removal of connection pools for versions was
ported from Zope 2.6. This feature is needed by Zope to avoid denial
of service attacks that allow a client to create an arbitrary number
of version pools.

A pair of new scripts from Jim Fulton can be used to synthesize
workloads and measure ZEO performance: see zodbload.py and
zeoserverlog.py in the Tools directory. Note that these require
Zope.

Tools/checkbtrees.py was strengthened in two ways:

In addition to running the _check() method on each BTree B found,
BTrees.check.check(B) is also run. The check() function was written
after checkbtrees.py, and identifies kinds of damage B._check()
cannot find.

Cycles in the object graph no longer lead to unbounded output.
Note that preventing this requires remembering the oid of each
persistent object found, which increases the memory needed by the
script.

If a storage server fails or times out between the vote and the
finish, the ZEO cache could get populated with objects that didn't
make it to the storage server.

If a client loses its connection to the server near the end of a
transaction, it is now guaranteed to get a ClientDisconnected error
even if it reconnects before the transaction finishes. This is
necessary because the server will always abort the transaction.
In some cases, the client would never see an error for the aborted
transaction.

In tpc_finish(), reordered the calls so that the server's tpc_finish()
is called (and must succeed) before we update the ZEO client cache.

The storage name is now prepended to the sort key, to ensure a
unique global sort order if storages are named uniquely. This
can prevent deadlock in some unusual cases.

A variety of fixes and improvements to Berkeley storage (aka BDBStorage)
were back-ported from ZODB 4. This release now contains the most
current version of the Berkeley storage code. Many tests have been
back-ported, but not all.

Modified the Windows tests to wait longer at the end of ZEO tests for
the server to shut down. Before Python 2.3, there is no waitpid() on
Windows, and, thus, no way to know if the server has shut down. The
change makes the Windows ZEO tests much less likely to fail or hang,
at the cost of increasing the time needed to run the tests.

Fixed a bug in ExtensionClass when comparing ExtensionClass instances.
The code could raise RuntimeWarning under Python 2.3, and produce
incorrect results on 64-bit platforms.

Fixed bugs in Tools/repozo.py, including a timing-dependent one that
could cause the following invocation of repozo to do a full backup when
an incremental backup would have sufficed.

Added Tools/README.txt that explains what each of the scripts in the
Tools directory does.

Fixed critical race conditions in ZEO's cache consistency code that
could cause invalidations to be lost or stale data to be written to
the cache. These bugs can lead to data loss or data corruption.
These bugs are relatively unlikely to be provoked in sites with few
conflicts, but the possibility of failure existed any time an object
was loaded and stored concurrently.

Fixed a bug in conflict resolution that failed to ghostify an object
if it was involved in a conflict. (This code may be redundant, but it
has been fixed regardless.)

The ZEO server was fixed so that it does not perform any I/O until all
of a transactions' invalidations are queued. If it performs I/O in the
middle of sending invalidations, it would be possible to overlap a
load from a client with the invalidation being sent to it.

The ZEO cache now handles invalidations atomically. This is the same
sort of bug that is described in the 3.1.2b1 section below, but it
affects the ZEO cache.

Fixed several serious bugs in fsrecover that caused it to fail
catastrophically in certain cases because it thought it had found a
checkpoint (status "c") record when it was in the middle of the file.

Invalidations are now processed atomically. Each transaction will see
all the changes caused by an earlier transaction or none of them.
Before this patch, it was possible for a transaction to see invalid
data because it saw only a subset of the invalidations. This is the
most likely cause of reported BTrees corruption, where keys were
stored in the wrong bucket. When a BTree bucket splits, the bucket
and the bucket's parent are both modified. If a transaction sees the
invalidation for the bucket but not the parent, the BTree in memory
will be internally inconsistent and keys can be put in the wrong
bucket. The atomic invalidation fix prevents this problem.

A number of minor reference count fixes in the object cache were
fixed. That's the cPickleCache.c file.

It was possible for a transaction that failed in tpc_finish() to lose
the traceback that caused the failure. The transaction code was fixed
to report the original error as well as any errors that occur while
trying to recover from the original error.

Trying to store an object of a non-integer type into an IIBTree
or OIBTree could leave the bucket in a variety of insane states. For
example, trying

b[obj] = "I'm a string, not an integer"

where b is an OIBTree. This manifested as a refcount leak in the test
suite, but could have been much worse (most likely in real life is that
a seemingly arbitrary existing key would "go missing").

When deleting the first child of a BTree node with more than one
child, a reference to the second child leaked. This could cause
the entire bucket chain to leak (not be collected as garbage
despite not being referenced anymore).

Comparing a Missing.Value object to a C type that provide its own
comparison operation could lead to a segfault when the Missing.Value
was on the right-hand side of the comparison operator. The Missing
class was fixed so that its coercion and comparison operations are
safe.

The Transaction "hosed" feature is disabled in this release. If a
transaction fails during the tpc_finish() it is not possible, in
general, to know whether the storage is in a consistent state. For
example, a ZEO server may commit the data and then fail before sending
confirmation of the commit to the client. If multiple storages are
involved in a transaction, the problem is exacerbated: One storage may
commit the data while another fails to commit. In previous versions
of ZODB, the database would set a global "hosed" flag that prevented
any other transaction from committing until an administrator could
check the status of the various failed storages and ensure that the
database is in a consistent state. This approach favors data
consistency over availability. The new approach is to log a panic but
continue. In practice, availability seems to be more important than
consistency. The failure mode is exceedingly rare in either case.

The BTrees-based fsIndex for FileStorage is enabled. This version of
the index is faster to load and store via pickle and uses less memory
to store keys. We had intended to enable this feature in an earlier
release, but failed to actually do it; thus, it's getting enabled as a
bug fix now.

Two rare bugs were fixed in BTrees conflict resolution. The most
probable symptom of the bug would have been a segfault. The bugs
were found via synthetic stress tests rather than bug reports.

A value-based consistency checker for BTrees was added. See the
module BTrees.check for the checker and other utilities for working
with BTrees.

A new script called repozo.py was added. This script, originally
written by Anthony Baxter, provides an incremental backup scheme for
FileStorage based storages.

zeopack.py has been fixed to use a read-only connection.

Various small autopack-related race conditions have been fixed in the
Berkeley storage implementations. There have been some table changes
to the Berkeley storages so any storage you created in 3.1.1b1 may not
work. Part of these changes was to add a storage version number to
the schema so these types of incompatible changes can be avoided in
the future.

The ZEO version number was bumped to 2.0.2 on account of the following
minor feature additions.

The performance of full cache verification has improved dramatically.
XXX Get measurements from Jim -- somewhere in 2x-5x recall. The
implementation was fixed to use the very-fast getSerial() method on
the storage instead of the comparatively slow load().

The ZEO server has an optional timeout feature that will abort a
connection that does not commit within a certain amount of time. The
timeout works by closing the socket the client is using, causing both
client and server to abort the transaction and continue. This is a
drastic step, but can be useful to prevent a hung client or other bug
from blocking a server indefinitely.

If a client was disconnected during a transaction, the tpc_abort()
call did not properly reset the internal state about the transaction.
The bug caused the next transaction to fail in its tpc_finish().
Also, any ClientDisconnected exceptions raised during tpc_abort() are
ignored.

ZEO logging has been improved by adding more logging for important
events, and changing the logging level for existing messages to a more
appropriate level (usually lower).

It was possible for earlier versions of ZODB to deadlock when using
multiple storages. If multiple transactions committed concurrently
and both transactions involved two or more shared storages, deadlock
was possible. This problem has been fixed by introducing a sortKey()
method to the transaction and storage APIs that is used to define an
ordering on transaction participants. This solution will prevent
deadlocks provided that all transaction participants that use locks
define a valid sortKey() method. A warning is raised if a participant
does not define sortKey(). For backwards compatibility, BaseStorage
provides a sortKey() that uses __name__.

Added code to ThreadedAsync/LoopCallback.py to work around a bug in
asyncore.py: a handled signal can cause unwanted reads to happen.

A bug in FileStorage related to object uncreation was fixed. If an
a transaction that created an object was undone, FileStorage could
write a bogus data record header that could lead to strange errors if
the object was loaded. An attempt to load an uncreated object now
raises KeyError, as expected.

The restore() implementation in FileStorage wrote incorrect
backpointers for a few corner cases involving versions and undo. It
also failed if the backpointer pointed to a record that was before the
pack time. These specific bugs have been fixed and new test cases
were added to cover them.

A bug was fixed in conflict resolution that raised a NameError when a
class involved in a conflict could not be loaded. The bug did not
affect correctness, but prevent ZODB from caching the fact that the
class was unloadable. A related bug prevented spurious
AttributeErrors when a class could not be loaded. It was also fixed.

The script Tools/zeopack.py was fixed to work with ZEO 2. It was
untested and had two silly bugs.

Some C extensions included standard header files before including
Python.h, which is not allowed. They now include Python.h first,
which eliminates compiler warnings in certain configurations.

The BerkeleyDB based storages have been merged from the trunk,
providing a much more robust version of the storages. They are not
backwards compatible with the old storages, but the decision was made
to update them in this micro release because the old storages did not
work for all practical purposes. For details, see Doc/BDBStorage.txt.

If an error occurs during conflict resolution, the store will silently
catch the error, log it, and continue as if the conflict was
unresolvable. ZODB used to behave this way, and the change to catch
only ConflictError was causing problems in deployed systems. There
are a lot of legitimate errors that should be caught, but it's too
close to the final release to make the substantial changes needed to
correct this.

A small extension was made to the iterator protocol. The Record
objects, which are returned by the per-transaction iterators, contain
a new data_txn attribute. It is None, unless the data contained in
the record is a logical copy of an earlier transaction's data. For
example, when transactional undo modifies an object, it creates a
logical copy of the earlier transaction's data. Note that this
provide a stronger statement about consistency than whether the data
in two records is the same; it's possible for two different updates to
an object to coincidentally have the same data.

The restore() method was extended to take the data_txn attribute
mentioned above as an argument. FileStorage uses the new argument to
write a backpointer if possible.

A few bugs were fixed.

The setattr slot of the cPersistence C API was being initialized to
NULL. The proper initialization was restored, preventing crashes in
some applications with C extensions that used persistence.

The return value of TimeStamp's __cmp__ method was clipped to return
only 1, 0, -1.

The restore() method was fixed to write a valid backpointer if the
update being restored is in a version.

Several bugs and improvements were made to zdaemon, which can be used
to run the ZEO server. The parent now forwards signals to the child
as intended. Pidfile handling was improved and the trailing newline
was omitted.

A few bugs have been fixed, some that were found with the help of
Neal Norwitz's PyChecker.

The zeoup.py tool has been fixed to allow connecting to a read-only
storage, when the --nowrite option is given.

Casey Duncan fixed a few bugs in the recent changes to undoLog().

The fstest.py script no longer checks that each object modified in a
transaction has a serial number that matches the transaction id.
This invariant is no longer maintained; several new features in the
3.1 release depend on it.

The ZopeUndo package was added. If ZODB3 is being used to run a ZEO
server that will be used with Zope, it is usually best if the server
and the Zope client don't share any software. The Zope undo
framework, however, requires that a Prefix object be passed between
client and server. To support this use, ZopeUndo was created to hold
the Prefix object.

Many bugs were fixed in ZEO, and a couple of features added. See
ZEO-NEWS.txt for details.

The ZODB guide included in the Doc directory has been updated. It is
still incomplete, but most of the references to old ZODB packages have
been removed. There is a new section that briefly explains how to use
BTrees.

The zeoup.py tool connects using a read-only connection when --nowrite
is specifified. This feature is useful for checking on read-only ZEO
servers.

Toby Dickenson implemented a new Connection cache for ZODB. The cache
is responsible for pointer swizzling (translating between oids and
Python objects) and for keeping recently used objects in memory. The
new cache is a big improvement over the old cache. It strictly honors
its size limit, where size is specified in number of objects, and it
evicts objects in least recently used (LRU) order.

Users should take care when setting the cache size, which has a
default value of 400 objects. The old version of the cache often held
many more objects than its specified size. An application may not
perform as well with a small cache size, because the cache no longer
exceeds the limit.

The index used by FileStorage was reimplemented using a custom BTrees
object. The index maps oids to file offsets, and is kept in memory at
all times. The new index uses about 1/4 the memory of the old,
dictionary-based index. See the module ZODB.fsIndex for details.

A security flaw was corrected in transactionalUndo(). The transaction
ids returned by undoLog() and used for transactionalUndo() contained a
file offset. An attacker could construct a pickle with a bogus
transaction record in its binary data, deduce the position of the
pickle in the file from the undo log, then submit an undo with a bogus
file position that caused the pickle to get written as a regular data
record. The implementation was fixed so that file offsets are not
included in the transaction ids.

Several storages now have an explicit read-only mode. For example,
passing the keyword argument read_only=1 to FileStorage will make it
read-only. If a write operation is performed on a read-only storage,
a ReadOnlyError will be raised.

The storage API was extended with new methods that support the Zope
Replication Service (ZRS), a proprietary Zope Corp product. We expect
these methods to be generally useful. The methods are:

restore(oid, serialno, data, version, transaction)

Perform a store without doing consistency checks. A client can
use this method to provide a new current revision of an object.
The serialno argument is the new serialno to use for the
object, not the serialno of the previous revision.

lastTransaction()

Returns the transaction id of the last committed transaction.

lastSerial(oid)

Return the current serialno for oid or None.

iterator(start=None, stop=None)

The iterator method isn't new, but the optional start and
stop arguments are. These arguments can be used to specify
the range of the iterator -- an inclusive range [start, stop].

FileStorage is now more cautious about creating a new file when it
believes a file does not exist. This change is a workaround for bug
in Python versions upto and including 2.1.3. If the interpreter was
builtin without large file support but the platform had it,
os.path.exists() would return false for large files. The fix is to
try to open the file first, and decide whether to create a new file
based on errno.

The undoLog() and undoInfo() methods of FileStorage can run
concurrently with other methods. The internal storage lock is
released periodically to give other threads a chance to run. This
should increase responsiveness of ZEO clients when used with ZEO 2.

New serial numbers are assigned consistently for abortVersion() and
commitVersion(). When a version is committed, the non-version data
gets a new serial number. When a version is aborted, the serial
number for non-version data does not change. This means that the
abortVersion() transaction record has the unique property that its
transaction id is not the serial number of the data records.

Berkeley storage constructors now take an optional config argument,
which is an instance whose attributes can be used to configure such
BerkeleyDB policies as an automatic checkpointing interval, lock table
sizing, and the log directory. See bsddb3Storage/BerkeleyBase.py for
details.

A getSize() method has been added to all Berkeley storages.

Berkeley storages open their environments with the DB_THREAD flag.

Some performance optimizations have been implemented in Full storage,
including the addition of a helper C extension when used with Python
2.2. More performance improvements will be added for the ZODB 3.1
final release.

A new experimental Autopack storage was added which keeps only a
certain amount of old revision information. The concepts in this
storage will be folded into Full and Autopack will likely go away in
ZODB 3.1 final. ZODB 3.1 final will also have much improved Minimal
and Full storages, which eliminate Berkeley lock exhaustion problems,
reduce memory use, and improve performance.

It is recommended that you use BerkeleyDB 4.0.14 and PyBSDDB 3.4.0
with the Berkeley storages. See bsddb3Storage/README.txt for details.

Tim Peters fixed several endcase bugs in the BTrees code. Most
importantly, after a mix of inserts and deletes in a BTree or TreeSet, it
was possible (but unlikely) for the internal state of the object to become
inconsistent. Symptoms then varied; most often this manifested as a
mysterious failure to find a key that you knew was present, or that
tree.keys() would yield an object that disgreed with the tree about
how many keys there were.

If you suspect such a problem, BTrees and TreeSets now support a ._check()
method, which does a thorough job of examining the internal tree pointers
for consistency. It raises AssertionError if it finds any problems, else
returns None. If ._check() raises an exception, the object is damaged,
and rebuilding the object is the best solution. All known ways for a
BTree or TreeSet object to become internally inconsistent have been
repaired.

Other fixes include:

Many fixes for range search endcases, including the "range search bug:"
If the smallest key S in a bucket in a BTree was deleted, doing a range
search on the BTree with S on the high end could claim that the range
was empty even when it wasn't.

The BTree module functions weightedIntersection() and weightedUnion()
now treat negative weights as documented. It's hard to explain what
their effects were before this fix, as the sign bits were getting
confused with an internal distinction between whether the result
should be a set or a mapping.

This version of ZODB ships with two different versions of ZEO. It
includes ZEO 2.0 beta 1, the recommended new version. (ZEO 2 will
reach final release before ZODB3.) The ZEO 2.0 protocol is not
compatible with ZEO 1.0, so we have also included ZEO 1.0 to support
people already using ZEO 1.0.

When a ConflictError is raised, the exception object now has a
sensible structure, thanks to a patch from Greg Ward. The exception
now uses the following standard attributes: oid, class_name, message,
serials. See the ZODB.POSException.ConflictError doc string for
details.

It is now easier to customize the registration of persistent objects
with a transaction. The low-level persistence mechanism in
cPersistence.c registers with the object's jar instead of with the
current transaction. The jar (Connection) then registers with the
transaction. This redirection would allow specialized Connections to
change the default policy on how the transaction manager is selected
without hacking the Transaction module.

Empty transactions can be committed without interacting with the
storage. It is possible for registration to occur unintentionally and
for a persistent object to compensate by making itself as unchanged.
When this happens, it's possible to commit a transaction with no
modified objects. The change allows such transactions to finish even
on a read-only storage.

Two new tools were added to the Tools directory. The analyze.py
script, based on a tool by Matt Kromer, prints a summary of space
usage in a FileStorage Data.fs. The checkbtrees.py script scans a
FileStorage Data.fs. When it finds a BTrees object, it loads the
object and calls the _check method. It prints warning messages
for any corrupt BTrees objects found.

This was the first public release of the StandaloneZODB from Zope
Corporation. Everything's new! :)

Copyright (c) 2011 Zope Foundation. All rights reserved.
Legal | Contact
If you can read this text, it means you are not experiencing the Plone design at its best.
Plone makes heavy use of CSS, which means it is accessible to any internet browser,
but the design needs a
standards-compliant browser to look like we intended it.
Just so you know ;)