A new option connections-pooler-enabled (default yes) has been added. This
allow to switch off internal connection pooling for use with others poolers
such as pgbouncer.

In deleteconf view (confirmation before deletion), the list of first-level
composite objects that would be deleted along with the primary entity is
displayed (01eeea97e549).

The cubicweb.pyramid module now provides a Paste application factory
registered as an entry point named pyramid_main and that can be used to
run a Pyramid WSGI application bound to a CubicWeb repository.

A new configuration type pyramid has been added to create CubicWeb’s
instances (through cubicweb-ctlcreate-cpyramid<basecube><appid>).
This configuration bootstraps a CubicWeb instance that is essentially a
repository plus the minimal setup to run a Pyramid WSGI application on top
of it. Noticeably, it does not ship all web configuration but rather
relies on configuration declared in a development.ini file for any
Pyramid application.

A new way to declare workflows as simple data structure (dict/list) has been
introduced. Respective utility functions live in cubicweb.wfutils
module. This handles both the creation and migration of workflows.

A new IDublinCore adapter has been introduced to control the generation of
Dublin Core metadata that are used in several base views.

It is now possible to derive rtags using their derive method
(0849a5eb57b8). Derived rtags keep a reference to the original rtag and only
hold custom rules, allowing changes which are done in the original rtag after
derivation to be still considered.

A new cubicweb-ctlscheduler<appid> command has been introduced to run
background and periodic tasks of the repository (previously called looping
tasks). In a production environment, a process with this command should be
run alongside with a WSGI server process (possibly running multiple
processes itself).

As a consequence of the replacement of the old looping tasks manager by a
scheduler, all cubicweb-ctl’s “start” commands (i.e. start, pyramid,
wsgi) do not start repository looping tasks manager anymore, nor do
they start the scheduler. Site administrators are thus expected to start
this scheduler as a separate process. Also, registering looping tasks (i.e.
calling repo.looping_tasks()) is a no-op when the repository has no
scheduler set; a warning is issued in such cases. Application developers may
rely on repository’s has_scheduler method to determine if they should
register a looping task or not.

Several cleanups in repository’s session management have been conducted
resulting from changes introduced in 3.19 release. Among others, the
cubicweb.server.session.Session class has been dropped, and request
session attribute is now tight to a web session whose implementation
depends on the front-end used (twisted or pyramid). Hence this attribute
should not be accessed from “repository side” code (e.g. hooks or operations)
and has lost some of his former attributes like repo which used to
reference the repository instance. Due to this, you don’t have anymore access
to session’s data through the connection, which leds to deprecation of the
data attribute and removal of get_shared_data and set_shared_data
methods which are deprecated since 3.19.

Support for ‘https-url’ configuration option has been removed
(4516c3956d46).

The next_tabindex method of request class has been removed (011730a4af73).

The cubicweb.hook.logstats.start hook was dropped because it’s looping
task would not be run in a web instance (see first point about repository
scheduler).

uicfg rules to hide the opposite relation of inlined form are not anymore
automatically added, because this was actually done randomly and so not
reliable, so you’ll have to add them manually:

Various bits of a CubicWeb application configuration can be now be
overridden through environments variables matching configuration option
names prefixed by CW_ (for instance CW_BASE_URL).

Cubes are now standard Python packages named as cubicweb_<cubename>.
They are not anymore installed in <prefix>/share/cubicweb/cubes. Their
discovery by CubicWeb is handled by a new setuptools entry point
cubicweb.cubes. A backward compatibility layer is kept for “legacy”
cubes.

Pyramid support made it into CubicWeb core. Applications that use it
should now declare the cubicweb[pyramid] dependency instead of
cubicweb-pyramid.

New NullStore class in cubicweb.dataimport.stores as new base class for
every store, and allowing to test your dataimport chain without actually
importing anything.

There has been several important changes to the core internals of CubicWeb:

Dropped asource and extid columns from the entities table as well as the
index on the type column, for a sensible optimization on both massive data
insertion and database size / index rebuilding.

Dropped the moved_entities table and related mecanism to remember that an
entity has been moved from a source to the system database - this is now the
responsability of source’s parser to detect this (usually by remembering its
original external id as cwuri).

Dropped the original ‘give me an eid for this extid, but call me back on
another line if it has to be created’ mecanism on which the cwxmlparser was
relying, in favor of parsers using the dataimport API. This includes dropping
the cwxmlparser. If you’re using it, you’ll have to write a specific
parser, examples to come.

Dropped source mapping handling (schema, views, logic) with its client
the cwxmlparser. This is not worth the burden, specific parsers should be
preferred.

The above changes lead to the following API changes:

req.entity_metas(eid) doesn’t return anymore a ‘type’ nor ‘source’ keys, use
req.entity_type(eid) instead or ‘cw_source’ relation to get those
information,

dropped ‘cw_source’ key from default JSON representation of an entity,

dropped source_uris() and handle_deletion(…) method from datafeed parser
base class, deletion of entities is now the responsability of specific
implementation (see ldapparser for example),

entities from external source are always displayed in the UI with a link
to the local entity, not the original one simplifying entity.absolute_url()
implementation and allowing to drop use_ext_eid argument of
entity.rest_path() (though it’s still supported for backward compat).

Python 3.x support in CubicWeb itself is now complete, except for the
twisted package (since Twisted does not completely support Python 3.x
itself). The skeleton for new cube should also be Python 3 compatible, in
particular its setup.py got updated.

The source-sync command can now synchronize all sources in the database,
if no <source> argument is provided.

Datafeed source synchronization is now asynchronous when requested from user
interface.

Most indexes and constraints will be rebuilt during the migration, because
they are now named after a md5 hash to control the name’s size.

Index are renamed upon renaming of an entity type, so they are still
correctly tracked.

A new db-check-index command is added to cubicweb-ctl, to display the
differences between the indexes in the database and those expected by the
schema. It’s recommended to run this command after the migration to 3.23 and
to adjust things manually for cases that are not easily handled by the
migration script, such as indexes of entity types that have been renamed. It
should be mostly about dropping extra indexes.

Generative tests à la logilab-common are not supported anymore in
CubicWebTC. It is advised to use the subtests API (available on
CubicWebTC either from the standard library as of Python 3.4 or through
unittest2 package otherwise).

CubicWebTC’s set_description method (comming from
logilab.common.testlib.TestCase) is no longer available.

When installed within a virtualenv, CubicWeb will look for instances data as
in user mode by default, that is in $HOME/etc/cubicweb.d, as opposed
to $VIRTUAL_ENV/etc/cubicweb.d previously. To restore this behavior,
explicitly set CW_MODE to system. Alternatively (and preferably), the
CW_INSTANCES_DIR environment variables may be used to specify instances
data location.

a huge amount of changes were done towards python 3.x support (as yet
incomplete). This introduces a new dependency on six, to handle
python2/python3 compatibility.

new cubicweb.dataimport.massive_store module, a postgresql-specific store
using the COPY statement to accelerate massive data imports. This
functionality was previously part of cubicweb-dataio (there are some API
differences with that previous version, however).

cubes custom sql scripts are executed before creating tables. This allows
them to create new types or extensions.

the ejsonexport view can be specialized using the new ISerializable
entity adapter. By default, it will return an entity’s (non-Bytes and
non-Password) attributes plus the special cw_etype and cw_source
keys.

cubes that define custom final types are now handled by the add_cube
migration command.

synchronization of external sources can be triggered from the web interface
by suitably privileged users with a new cw.source-sync action.

the ldapfeed source now depends on the ldap3 module instead of
python-ldap.

replies don’t get an Expires header by default. However when they do,
they also get a coherent Cache-Control.

data files are regenerated at each request, they are no longer cached by
cubicweb.web.PropertySheet. Requests for data files missing the instance
hash are handled with a redirection instead of a direct reply, to allow
correct cache-related reply headers.

config.repository() creates a new Repository object each time, instead of
returning a cached object. WARNING: this may cause unexpected issues if
several repositories end up being used.

migration scripts, as well as other scripts executed by cubicweb-ctlshell, are loaded with the print_function flag enabled (for backwards
compatibility, if that fails they are re-loaded without that flag)

the cw_fti_index_rql_queries method on entity classes is replaced by
cw_fti_index_rql_limit, a generator which yields ResultSet objects
containing entities to be indexed. By default, entities are returned 1000 at
a time.

the Repository.extid2eid() entry point for external sources is deprecated.
Imports should use one of the stores from the cubicweb.dataimport package
instead.

the cubicweb.repoapi.get_repository() function’s uri argument should
no longer be used.

the generic datafeed xml parser is deprecated in favor of the “store” API
introduced in cubicweb 3.21.

the session manager lives in the sessions registry instead of components.

TZDatetime attributes are returned as timezone-aware python datetime
objects. WARNING: this will break client applications that compare or use
arithmetic involving timezone-naive datetime objects.

creation_date and modification_date attributes for all entities are now
timezone-aware (TZDatetime) instead of localtime (Datetime). More
generally, the Datetime type should be considered as deprecated.

req.set_cookie’s “expires” argument, if not None, is expected to be a
date or a datetime in UTC. It was previously interpreted as localtime
with the UTC offset the server started in, which was inconsistent (we
are not aware of any users of that API).

the way to run tests on a postgresql backend has changed slightly, use
cubicweb.devtools.{start,stop}pgcluster in setUpModule and tearDownModule

the Connection and ClientConnection objects introduced in CubicWeb 3.19 have
been unified. To connect to a repository, use:

the user_callback api has been removed; people should use plain
ajax functions instead

the Pyro and Zmq-pickle remote repository access methods have
been entirely removed (emerging alternatives such as rqlcontroller
and cwclientlib should be used instead). Note that as a side effect,
“repository-only” instances (i.e. without a http component) are no
longer possible. If you have any such instances, you will need to
rename the configuration file from repository.conf to all-in-one.conf
and run cubicweb-ctlupgrade to update it. Likewise, remote cubicweb-ctl
shell is no longer available.

ucsvreader() and ucsvreader_pb() from the dataimport module have
2 new keyword arguments delimiter and quotechar to replace the
separator and quote arguments respectively. This makes the API match
that of Python’s csv.reader(). The old arguments are still supported
though deprecated.

the migration environment’s remove_cube function is now called drop_cube.

cubicweb.old.css is now cubicweb.css. The previous “new”
cubicweb.css, along with its cubicweb.reset.css companion, have been
removed.

The anonymous property of Session and Connection are now computed from the
related user login. If it matches the anonymous-user in the config the
connection is anonymous. Beware that the anonymous-user config is web
specific. Therefore, no session may be anonymous in a repository only setup.

A new explicit Connection object replaces Session as the main repository entry
point. Connection holds all the necessary methods to be used server-side
(execute, commit, rollback, call_service, entity_from_eid,
etc…). One obtains a new Connection object using session.new_cnx().
Connection objects need to have an explicit begin and end. Use them as a context
manager to never miss an end:

withsession.new_cnx()ascnx:cnx.execute('INSERT Elephant E, E name "Babar"')cnx.commit()cnx.execute('INSERT Elephant E, E name "Celeste"')cnx.commit()# Once you get out of the "with" clause, the connection is closed.

Using the same Connection object in multiple threads will give you access to the
same Transaction. However, Connection objects are not thread safe (hence at your
own risks).

repository.internal_session is deprecated in favor of
repository.internal_cnx. Note that internal connections are now safe by default,
i.e. the integrity hooks are enabled.

A new API has been introduced to replace the dbapi. It is called repoapi.

There are three relevant functions for now:

repoapi.get_repository returns a Repository object either from an
URI when used as repoapi.get_repository(uri) or from a config
when used as repoapi.get_repository(config=config).

repoapi.connect(repo,login,**credentials) returns a ClientConnection
associated with the user identified by the credentials. The
ClientConnection is associated with its own Session that is closed
when the ClientConnection is closed. A ClientConnection is a
Connection-like object to be used client side.

repoapi.anonymous_cnx(repo) returns a ClientConnection associated
with the anonymous user if described in the config.

On the client/web side, the Request is now using a repoapi.ClientConnection
instead of a dbapi.connection. The ClientConnection has multiple backward
compatible methods to make it look like a dbapi.Cursor and dbapi.Connection.

Session used on the Web side are now the same than the one used Server side.
Some backward compatibility methods have been installed on the server side Session
to ease the transition.

The authentication stack has been altered to use the repoapi instead of
the dbapi. Cubes adding new element to this stack are likely to break.

Session data can be accessed using the cnx.data dictionary, while
transaction data is available through cnx.transaction_data. These
replace the [gs]et_shared_data methods with optional txid kwarg.

All current methods and attributes used to access the repo on CubicWebTC are
deprecated. You may now use a RepoAccess object. A RepoAccess object is
linked to a new Session for a specified user. It is able to create
Connection, ClientConnection and web side requests linked to this
session:

access=self.new_access('babar')# create a new RepoAccess for user babarwithaccess.repo_cnx()ascnx:# some work with server side cnxcnx.execute(...)cnx.commit()cnx.execute(...)cnx.commit()withaccess.client_cnx()ascnx:# some work with client side cnxcnx.execute(...)cnx.commit()withaccess.web_request(elephant='babar')asreq:# some work with client side cnxelephant_name=req.form['elephant']req.execute(...)req.cnx.commit()

By default testcase.admin_access contains a RepoAccess object for the
default admin session.

RepositorySessionManager.postlogin is now called with two arguments,
request and session. And this now happens before the session is linked to the
request.

SessionManager and AuthenticationManager now take a repo object at
initialization time instead of a vreg.

The async argument of _cw.call_service has been dropped. All calls are
now synchronous. The zmq notification bus looks like a good replacement for
most async use cases.

repo.stats() is now deprecated. The same information is available through
a service (_cw.call_service('repo_stats')).

repo.gc_stats() is now deprecated. The same information is available through
a service (_cw.call_service('repo_gc_stats')).

repo.register_user() is now deprecated. The functionality is now
available through a service (_cw.call_service('register_user')).

request.set_session no longer takes an optional user argument.

CubicwebTC does not have repo and cnx as class attributes anymore. They are
standard instance attributes. set_cnx and _init_repo class methods
become instance methods.

set_cnxset and free_cnxset are deprecated. cnxset are now
automatically managed.

The implementation of cascading deletion when deleting composite
entities has changed. There comes a semantic change: merely deleting
a composite relation does not entail any more the deletion of the
component side of the relation.

_cw.user_callback and _cw.user_rql_callback are deprecated. Users
are encouraged to write an actual controller (e.g. using ajaxfunc)
instead of storing a closure in the session data.

A new entity.cw_linkable_rql method provides the rql to fetch all entities
that are already or may be related to the current entity using the given
relation.

introduce an add permission on attributes, to be interpreted at
entity creation time only and allow the implementation of complex
update rules that don’t block entity creation (before that the
update attribute permission was interpreted at entity creation and
update time)

the primary view display controller (uicfg) now has a
set_fields_order method similar to the one available for forms

new method ResultSet.one(col=0) to retrive a single entity and enforce the
result has only one row (see #3352314)

new method RequestSessionBase.find to look for entities
(see #3361290)

the embedded jQuery copy has been updated to version 1.10.2, and jQuery UI to
version 1.10.3.

initial support for wsgi for the debug mode, available through the new
wsgi cubicweb-ctl command, which can use either python’s builtin
wsgi server or the werkzeug module if present.

a rql-table directive is now available in ReST fields

cubicweb-ctl upgrade can now generate the static data resource directory
directly, without a manual call to gen-static-datadir.

not really an API change, but the entity permission checks are now
systematically deferred to an operation, instead of a) trying in a
hook and b) if it failed, retrying later in an operation

The default value storage for attributes is no longer String, but
Bytes. This opens the road to storing arbitrary python objects, e.g.
numpy arrays, and fixes a bug where default values whose truth value
was False were not properly migrated.

symmetric relations are no more handled by an rql rewrite but are
now handled with hooks (from the activeintegrity category); this
may have some consequences for applications that do low-level database
manipulations or at times disable (some) hooks.

unique together constraints (multi-columns unicity constraints)
get a name attribute that maps the CubicWeb contraint entities to
corresponding backend index.

BreadCrumbEntityVComponent’s open_breadcrumbs method now includes
the first breadcrumbs separator

entities can be compared for equality and hashed

the on_fire_transition predicate accepts a sequence of possible
transition names

the GROUP_CONCAT rql aggregate function no longer repeats duplicate
values, on the sqlite and postgresql backends

Add a new dataimport store (SQLGenObjectStore). This store enables a fast
import of data (entity creation, link creation) in CubicWeb, by directly
flushing information in SQL. This may only be used with PostgreSQL, as it
requires the ‘COPY FROM’ command.

Dropped ‘pyro-ns-host’, ‘pyro-instance-id’, ‘pyro-ns-group’ from the client side
configuration, in favor of ‘repository-uri’. NO MIGRATION IS DONE,
supposing there is no web-only configuration in the wild.

Stop discovering the connection method through repo_method class attribute
of the configuration, varying according to the configuration class. This is
a first step on the way to a simpler configuration handling.

DB-API related changes:

Stop indicating the connection method using ConnectionProperties.

Drop _cnxtype attribute from Connection and cnxtype from
Session. The former is replaced by a is_repo_in_memory property
and the later is totaly useless.

Turn repo_connect into _repo_connect to mark it as a private function.

Deprecate in_memory_cnx which becomes useless, use _repo_connect instead
if necessary.

the “tcp://” uri scheme used for ZMQ
communications (in a way reminiscent of Pyro) is now named
“zmqpickle-tcp://”, so as to make room for future zmq-based lightweight
communications (without python objects pickling).

The RQL search bar has now some auto-completion support. It means
relation types or entity types can be suggested while typing. It is
an awesome improvement over the current behaviour !

The action box associated with table views (from tableview.py)
has been transformed into a nice-looking series of small tabs; it
means that the possible actions are immediately visible and need not
be discovered by clicking on an almost invisible icon on the upper
right.

The uicfg module has moved to web/views/ and ui configuration
objects are now selectable. This will reduce the amount of
subclassing and whole methods replacement usually needed to
customize the ui behaviour in many cases.

Add Zmq server, based on the cutting edge ZMQ (http://www.zeromq.org/) socket
library. This allows to access distant instance, in a similar way as Pyro.

Publish/subscribe mechanism using ZMQ for communication among cubicweb
instances. The new zmq-address-sub and zmq-address-pub configuration variables
define where this communication occurs. As of this release this mechanism is
used for entity cache invalidation.

Improved WSGI support. While there is still some caveats, most of the code
which was twisted only is now generic and allows related functionalities to work
with a WSGI front-end.

Full undo/transaction support : undo of modification has eventually been
implemented, and the configuration simplified (basically you activate it or not
on an instance basis).

Controlling HTTP status code used is not much more easier :

WebRequest now has a status_out attribut to control the response status ;

The base registry implementation has been moved to a new
logilab.common.registry module (see #1916014). This includes code from :

cubicweb.vreg (the whole things that was in there)

cw.appobject (base selectors and all).

In the process, some renaming was done:

the top level registry is now RegistryStore (was VRegistry), but that
should not impact cubicweb client code ;

former selectors functions are now known as “predicate”, though you still use
predicates to build an object’selector ;

for consistency, the objectify_selector decoraror has hence be renamed to
objectify_predicate ;

on the CubicWeb side, the selectors module has been renamed to
predicates.

Debugging refactoring dropped the more need for the lltrace decorator. There
should be full backward compat with proper deprecation warnings. Notice the
yes predicate and objectify_predicate decorator, as well as the
traced_selection function should now be imported from the
logilab.common.registry module.

All login forms are now submitted to <app_root>/login. Redirection to requested
page is now handled by the login controller (it was previously handle by the
session manager).

Publisher.publish has been renamed to Publisher.handle_request. This
method now contains generic version of logic previously handled by
Twisted. Controller.publish is not affected.

New ‘zmqrql’ source type, similar to ‘pyrorql’ but using ømq instead of Pyro.

A new registry called services has appeared, where you can register
server-side cubicweb.server.Service child classes. Their call method can be
invoked from a web-side AppObject instance using new self._cw.call_service
method or a server-side one using self.session.call_service. This is a new
way to call server-side methods, much cleaner than monkey patching the
Repository class, which becomes a deprecated way to perform similar tasks.

a new ajax-func registry now hosts all remote functions (i.e. functions
callable through the asyncRemoteExec JS api). A convenience ajaxfunc
decorator will let you expose your python function easily without all the
appobject standard boilerplate. Backward compatibility is preserved.

the ‘json’ controller is now deprecated in favor of the ‘ajax’ one.

WebRequest.build_url can now take a __secure__ argument. When True cubicweb
try to generate an https url.

Entity.fetch_rqlrestriction argument has been deprecated and should be
replaced with a call to the new Entity.fetch_rqlst method, get the returned
value (a rql Select node) and use the RQL syntax tree API to include the
above-mentionned restrictions.

Backward compat is kept with proper warning.

Entity.fetch_order and Entity.fetch_unrelated_order class methods have been
replaced by Entity.cw_fetch_order and Entity.cw_fetch_unrelated_order with
a different prototype:

instead of taking (attr, var) as two string argument, they now take (select,
attr, var) where select is the rql syntax tree beinx constructed and var the
variable node.

instead of returning some string to be inserted in the ORDERBY clause, it has
to modify the syntax tree

Backward compat is kept with proper warning, BESIDE cases below:

custom order method return something else the a variable name with or
without the sorting order (e.g. cases where you sort on the value of a
registered procedure as it was done in the tracker for instance). In such
case, an error is logged telling that this sorting is ignored until API
upgrade.

client code use direct access to one of those methods on an entity (no code
known to do that).

Entity._rest_attr_info class method has been renamed to
Entity.cw_rest_attr_info

No backward compat yet since this is a protected method an no code is known to
use it outside cubicweb itself.

AnyEntity.linked_to has been removed as part of a refactoring of this
functionality (link a entity to another one at creation step). It was replaced
by a EntityFieldsForm.linked_to property.

In the same refactoring, cubicweb.web.formfield.relvoc_linkedto,
cubicweb.web.formfield.relvoc_init and
cubicweb.web.formfield.relvoc_unrelated were removed and replaced by
RelationField methods with the same names, that take a form as a parameter.

No backward compatibility yet. It’s still time to cry for it.
Cubes known to be affected: tracker, vcsfile, vcreview.

CWPermission entity type and its associated require_permission relation type
(abstract) and require_group relation definitions have been moved to a new
localperms cube. With this have gone some functions from the
cubicweb.schemas package as well as some views. This makes cubicweb itself
smaller while you get all the local permissions stuff into a single,
documented, place.

Backward compat is kept for existing instances, though you should have
installed the localperms cubes. A proper error should be displayed when
trying to migrate to 3.14 an instance the use CWPermission without the new
cube installed. For new instances / test, you should add a dependancy on the
new cube in cubes using this feature, along with a dependancy on cubicweb >=
3.14.

jQuery has been updated to 1.6.4 and jquery-tablesorter to 2.0.5. No backward
compat issue known.

Table views refactoring : new RsetTableView and EntityTableView, as well as
rewritten an enhanced version of PyValTableView on the same bases, with logic
moved to some column renderers and a layout. Those should be well documented
and deprecates former TableView, EntityAttributesTableView and CellView,
which are however kept for backward compat, with some warnings that may not be
very clear unfortunatly (you may see your own table view subclass name here,
which doesn’t make the problem that clear). Notice that _cw.view(‘table’,
rset, *kwargs) will be routed to the new RsetTableView or to the old
TableView depending on given extra arguments. See #1986413.

display_name don’t call .lower() anymore. This may leads to changes in your
user interface. Different msgid for upper/lower cases version of entity type
names, as this is the only proper way to handle this with some languages.

IEditControlAdapter has been deprecated in favor of EditController
overloading, which was made easier by adding dedicated selectors called
match_edited_type and match_form_id.

Pre 3.6 API backward compat has been dropped, though data migration
compatibility has been kept. You may have to fix errors due to old API usage
for your instance before to be able to run migration, but then you should be
able to upgrade even a pre 3.6 database.

Deprecated cubicweb.web.views.iprogress in favor of new iprogress cube.

Datafeed source now present an history of the latest import’s log, including
global status and debug/info/warning/error messages issued during
imports. Import logs older than a configurable amount of time are automatically
deleted.

Breadcrumbs component is properly kept when creating an entity with ‘__linkto’.

users and groups management now really lead to that (i.e. includes groups
management).