making a JSONP call to the CKAN API returns the wrong mime type (#4022)

Deleting a resource sets datastore_active=False to all resources and overrides their extras (#4042)

Deleting first Group and Organization custom field is not possible (#4094)

Changes and deprecations:

The default templates included in CKAN core have been updated to use Bootstrap 3. Extensions
implementing custom themes are encouraged to update their templates, but they can still
make CKAN load the old Bootstrap 2 templates during the transition using the following
configuration options:

As with all patch releases this one does not include requirement changes.
However in some scenarios you might encounter the following error while
installing or upgrading this version of CKAN:

Error:couldnotdeterminePostgreSQLversionfrom'10.2'

This is due to a bug in the psycopg2 version pinned to the release. To solve
it, upgrade psycopg2 with the following command:

pipinstall--upgradepsycopg2==2.7.3.2

This release does not require a Solr schema upgrade, but if you are having the
issues described in #3863 (datasets wrongly indexed in multilingual setups),
you can upgrade the Solr schema and reindex to solve them.

#3422 (implemented in #3425) introduced a major bug where if a resource was
deleted and the DataStore was active extras from all resources on the site where
changed. This is now fixed as part of this release but if your database is already
affected you will need to run a script to restore the extras to their
previous state. Remember, you only need to run the script if all the following are
true:

You are currently running CKAN 2.7.0 or 2.7.2, and

You have enabled the DataStore, and

One or more resources with data on the DataStore have been deleted (or you
suspect they might have been)

If all these are true you can run the following script to restore the extras to
their previous state:

organization_list_for_user (and the h.organizations_available()
helper) now return all organizations a user belongs to regardless of
capacity (Admin, Editor or Member), not just the ones where she is an
administrator (#2457)

organization_list_for_user (and the h.organizations_available()
helper) now default to not include package_count. Pass
include_dataset_count=True if you need the package_count values.

The API versions 1 and 2 (also known as the REST API, ie /api/rest/* will removed
in favour of the version 3 (action API, /api/action/*), which was introduced in
CKAN 2.0. The REST API will be removed on CKAN 2.8.

The default theme included in CKAN core will switch to use Bootstrap 3 instead of
Bootstrap 2 in CKAN 2.8. The current Bootstrap 2 based templates will still be included
in the next CKAN versions, so existing themes will still work. Bootstrap 2 templates will
be eventually removed though, so instances are encouraged to update their themes using
the available documentation (https://getbootstrap.com/migration/)

The activity stream related actions ending with *_list (eg package_activity_list)
and *_html (eg package_activity_list_html) will be removed in CKAN 2.8 in favour of
more efficient alternatives and are now deprecated.

The legacy revisions controller (ie /revisions/*) will be completely removed in CKAN 2.8.

The library used to render markdown has been changed to python-markdown. This
introduces both python-markdown and bleach as dependencies, as bleach
is used to clean any HTML provided to the markdown processor.

This is the last version of CKAN to support Postgresql 8.x, 9.0 and 9.1. The
next minor version of CKAN will require Postgresql 9.2 or later.

where returning extra additional fields by default, causing performance
issues. This is now fixed, so the output for these actions no longer returns
users, extras, etc.
Also, on the homepage template the c.groups and c.group_package_stuff
context variables are no longer available.

Bug fixes:

Fix dataset count in templates and show datasets on featured org/group (#2557)

Fix autodetect for TSV resources (#2553)

Improve character escaping in DataStore parameters

Fix “paster db init” when celery is configured with a non-database backend

The default of allowing anyone to create datasets, groups and organizations
has been changed to False. It is advised to ensure you set all of the
Authorization Settings options explicitly in your CKAN config. (#2164)

The package_show API call does not return the tracking_summary,
keys in the dataset or resources by default any more.

Any custom templates or users of this API call that use these values will
need to pass: include_tracking=True.

The legacy tests directory has moved to tests/legacy, the
new_tests directory has moved to tests and the new_authz.py
module has been renamed authz.py. Code that imports names from the
old locations will continue to work in this release but will issue
a deprecation warning. (#1753)

group_show and organization_show API calls no longer return the
datasets by default (#2206)

Custom templates or users of this API call will need to pass
include_datasets=True to include datasets in the response.

The vocabulary_show and tag_show API calls no longer returns the
packages key - i.e. datasets that use the vocabulary or tag.
However tag_show now has an include_datasets option. (#1886)

Config option site_url is now required - CKAN will not abort during
start-up if it is not set. (#1976)

Changes on the authentication mechanism to allow more secure setups (httponly
and secure cookies, disable CORS, etc). (#2004. #2050, #2052
…) See “Changes and deprecations” section for more details and
“Troubleshooting” for migration instructions.

By convention, view plugin names now end with _view rather than
_preview (eg recline_view rather than recline_preview). You will
need to update them on the ckan.plugins setting.

The way resource visualizations are created by default has changed. You might
need to set the ckan.views.default_views configuration option and run
a migration command on existing instances. Please refer to the migration
guide for more details:

The PDF Viewer extension has been moved to a separate extension:
https://github.com/ckan/ckanext-pdfview. Please install it separately if
you are using the pdf_view plugin (or the old pdf_preview one).

The action API (v3) no longer returns the full help for the action on each
request. It rather includes a link to a separate call to get the action
help string.

The user_show API call does not return the datasets,
num_followers or activity keys by default any more.

Any custom templates or users of this API call that use these values will
need to specify parameters: include_datasets or
include_num_followers.

activity has been removed completely as it was actually a list of
revisions, rather than the activity stream. If you want the actual activity
stream for a user, call user_activity_list instead.

The output of resource_show now contains a package_id key that links
to the parent dataset.

helpers.get_action() (or h.get_action() in templates) is deprecated.

Since action functions raise exceptions and templates cannot catch
exceptions, it’s not a good idea to call action functions from templates.

Instead, have your controller method call the action function and pass the
result to your template using the extra_vars param of render().

Alternatively you can wrap individual action functions in custom template
helper functions that handle any exceptions appropriately, but this is likely
to make your the logic in your templates more complex and templates are
difficult to test and debug.

Note that logic.get_action() and toolkit.get_action() are not deprecated,
core code and plugin code should still use get_action().

Cross-Origin Resource Sharing (CORS) support is no longer enabled by
default. Previously, Access-Control-Allow-* response headers were added for
all requests, with Access-Control-Allow-Origin set to the wildcard value
*. To re-enable CORS, use the new ckan.cors configuration settings
(ckan.cors.origin_allow_all and ckan.cors.origin_whitelist).

The HttpOnly flag will be set on the authorization cookie by default. For
enhanced security, we recommend using the HttpOnly flag, but this behaviour
can be changed in the Repoze.who settings detailed in the Config File
Options documentation (who.httponly).

The OpenID login option has been removed and is no longer supported. See
“Troubleshooting” if you are upgrading an existing CKAN instance as you may
need to update your who.ini file.

Note to people with custom themes: If you’ve changed the
{%blocksecondary_content%} in templates/package/search.html pay close
attention as this pull request changes the structure of that template block a
little.

Also: There’s a few more bootstrap classes (especially for grid layout) that
are now going to be in the templates. Take a look if any of the following
changes might effect your content blocks:

Note: This version does not require a requirements upgrade on source installations

Note: This version requires a database upgrade

Note: This version requires a Solr schema upgrade (The Solr schema file has
been renamed, the schema file from the previous release is compatible
with this version, but users are encouraged to point to the new one,
see “API changes and deprecations”)

Major:

Brand new automatic importer of tabular data to the DataStore, the
DataPusher. This is much more robust and simple to deploy and maintain than
its predecesor (ckanext-datastorer). Whole new UI for re-importing data to
the DataStore and view the import logs (#932, #938, #940, #981, #1196, #1200
…)

Completely revamped file uploads that allow closer integration with resources
and the DataStore, as well as making easir to integrate file uploads in other
features. For example users can now upload images for organizations and
groups. See “API changes and deprecations” if you are using the current
FileStore. (#1273, #1173 … )

UI and API endpoints for resource reordering (#1277)

Backend support for organization hierarchy, allowing parent and children
organizations. Frontend needs to be implemented in extensions (#1038)

User invitations: it is now possible to create new users with just their
email address. An invite email is sent to them, allowing to change their user
name and password (#1178)

Disable user registration with a configuration option (#1226)

Great effort in improving documentation, specially for customizing CKAN, with
a complete tutorial for writing extensions and customizing the theme. User
and sysadmin guides have also been moved to the main documentation
(#943, #847, #1253)

Minor:

Homepage modules to allow predefined layouts (#1126)

Ability to delete users (#1163)

Dedicated dataset groups page for displaying and managing them (#1102)

Implement organization_purge and group_purge action functions (#707)

Improve package_show performance (#1078)

Support internationalization of rendered dates and times (#1041)

Improve plugin load handling (#549)

Authorization function auditing for action functions (#1060)

Improve datetime rendering (#518)

New SQL indexes to improve performance (#1164)

Changes in requirements management (#1149)

Add offset/limit to package_list action (#1179)

Document all available configuraton options (#848)

Make CKAN sqlalchemy 0.8.4 compatible (#1427)

UI labelling and cleanup (#1030)

Better UX for empty groups/orgs (#1094)

Improve performance of group_dictize when the group has a lot of packages
(#1208)

Visibility field value not kept if there are errors on the form (#1265)

package_list should not return private datasets (#1295)

Fix 404 on organization activity stream and about page (#1298)

Fix placeholder images broken on non-root locations (#1309)

“Add Dataset” button shown on org pages when not authorized (#1348)

Fix exception when visiting organization history page (#1359)

Fix search ordering on organization home page (#1368)

datastore_search_sql failing for some anonymous users (#1373)

related_list logic function throws a 503 without any parameters (#1384)

Disabling activity_streams borks editing groups and user (#1421)

Member Editing Fixes (#1454)

Bulk editing broken in IE7 (#1455)

Fix group deletion in IE7 (#1460)

And many, many more!

API changes and deprecations:

The Solr schema file is now always named schema.xml regardless of the
CKAN version. Old schema files have been kept for backwards compatibility
but users are encouraged to point to the new unified one (#1314)

The FileStore and file uploads have been completely refactored and simplified
to only support local storage backend. The links from previous versions of
the FileStore to hosted files will still work, but there is a command
available to migrate the files to new Filestore. See this page for more
details:
http://docs.ckan.org/en/latest/filestore.html#filestore-21-to-22-migration

By default, the authorization for any action defined from an extension will
require a logged in user, otherwise a ckan.logic.NotAuthorized
exception will be raised. If an action function allows anonymous access (eg
search, show status, etc) the auth_allow_anonymous_access decorator
(available on the plugins toolkit) must be used (#1210)

package_search now returns results with custom schemas applied like
package_show, a use_default_schema parameter was added to request the
old behaviour, this change may affect customized search result templates
(#1255)

The ckan.api_url configuration option has been completely removed and it
can no longer be used (#960)

The edit and after_update methods of IPackageController plugins are now
called when updating a resource using the web frontend or the
resource_update API action (#1052)

Dataset moderation has been deprecated, and the code will probably be removed
in later CKAN versions (#1139)

One of the database changes on this version is the addition of a
metadata_modified field in the package table, that was filled during the
DB migration process. If you have previously migrated the database and revert
to an older CKAN version the migration process may have failed at this step,
leaving the fields empty. Also make sure to restart running processes like
harvesters after the update to make sure they use the new code base.

Note: This version requires a requirements upgrade on source installations

Note: This version requires a database upgrade

Note: This version does not require a Solr schema upgrade

Note

The json_preview plugin has been renamed to text_preview
(see #266). If you are upgrading CKAN from a previous version you need
to change the plugin name on your CKAN config file after upgrading to avoid
a PluginNotFound exception.

Major:

Bulk updates of datasets within organizations (delete, make public/private) (#278)

CKAN’s new “organizations” feature replaces the old authorization system
with a new one based on publisher organizations. It replaces the “Publisher
Profile and Workflow” feature from CKAN 1.X, any instances relying on it will
need to be updated.

CKAN’s frontend has been completely redesigned, inside and out. There is
a new default theme and the template engine has moved from Genshi to
Jinja2. Any custom templates using Genshi will need to be updated, although
there is a ckan.legacy_templates setting to aid in the migration.

New paster front-end-build command does everything needed to build the
frontend for a production CKAN site (runs paster less to compile the css
files, paster minify to minify the css and js files, etc.)

Plugins & Extensions:

New plugins toolkit provides a stable set of utility and helper functions
for CKAN plugins to depend on.

The IDatasetForm plugin interface has been redesigned (note: this breaks
backwards-compatibility with existing IDatasetForm plugins) (#649)

Many IDatasetForm bugs were fixed

New example extensions in core, and better documentation for the relevant
plugin interfaces: example_itemplatehelpers (#447),
example_idatasetform (#2750), hopefully more to come in 2.1!

New IFacets interface that allows to modify the facets shown on various
pages. (#400)

The get_action() function now automatically adds ‘model’ and ‘session’ to
the context dict (this saves on boiler-plate code, and means plugins don’t
have to import ckan.model in order to call get_action()) (#172)

Activity Streams, Following & User Dashboard:

New visual design for activity streams (#2941)

Group activity streams now include activities for changes to any of the
group’s datasets (#1664)

Group activity streams now appear on group pages (previously they could
only be retrieved via the api)

Dataset activity streams now appear on dataset pages (previously they could
only be retrieved via the api) (#3024)

Users can now follow groups (previously you could only follow users or
datasets) (#3005)

Activity streams and following are also supported for organizations (#505)

When you’re logged into CKAN, you now get a notifications count in the
top-right corner of the site, telling you how many new notifications you
have on your dashboard. Clicking on the count takes you to your dashboard
page to view your notifications. (#3009)

Optionally, you can also receive notifications by email when you have new
activities on your dashboard (#1635)

Infinite scrolling of activity streams (if you scroll to the bottom of a
an activity stream, CKAN will automatically load more activities) (#3018)

Redesigned user dashboard (#3028):

New dropdown-menu enables you to filter you dashboard activity stream to
show only activities from a particular user, dataset, group or
organization that you’re following

New sidebar shows previews and unfollow buttons (when the activity stream
is filtered)

CKAN’s DataStore and Data API have been rewritten, and now use PostgreSQL
instead of elasticsearch, so there’s no need to install elasticsearch
anymore (this feature was also back-ported to CKAN 1.8) (#2733)

New Config page for sysadmins (/ckan-admin/config) enables sysadmins to set
the site title, tag line, logo, the intro text shown on the front page,
the about text shown on the /about page, select a theme, and add custom
CSS (#2302, #2781)

Note: This version requires a requirements upgrade on source installations

Note: This version requires a database upgrade

Note: This version does not require a Solr schema upgrade

Major

New ‘follow’ feature that allows logged in users to follow other users or
datasets (#2304)

New user dashboard that shows an activity stream of all the datasets and
users you are following. Thanks to Sven R. Kunze for his work on this (#2305)

New version of the Datastore. It has been completely rewritten to use
PostgreSQL as backend, it is more stable and fast and supports SQL queries
(#2733)

Clean up and simplifyng of CKAN’s dependencies and source install
instructions. Ubuntu 12.04 is now supported for source installs (#2428,#2592)

Big speed improvements when indexing datasets (#2788)

New action API reference docs, which individually document each function and
its arguments and return values (#2345)

Updated translations, added Japanese and Korean translations

Minor

Add source install upgrade docs (#2757)

Mark more strings for translation (#2770)

Allow sort ordering of dataset listings on group pages (#2842)

Reenable simple search option (#2844)

Editing organization removes all datasets (#2845)

Accessibility enhancements on templates

Bug fixes

Fix for relative url being used when doing file upload to local storage

Various fixes on IGroupFrom (#2750)

Fix group dataset sort (#2722)

Fix adding existing datasets to organizations (#2843)

Fix 500 error in related controller (#2856)

Fix for non-open licenses appearing open

Editing organization removes all datasets (#2845)

API changes and deprecation:

Template helper functions are now restricted by default. By default only
those helper functions listed in lib.helpers.__allowed_functions__
are available to templates. The full functions can still be made
available by setting ckan.restrict_template_vars = false in your ini file.
Only restricted functions will be allowed in future versions of CKAN.

Deprecated functions related to the old faceting data structure have
been removed: helpers.py:facet_items(), facets.html:facet_sidebar(),
facets.html:facet_list_items().
Internal use of the old facets datastructure (attached to the context,
c.facets) has been superseded by use of the improved facet data structure,
c.search_facets. The old data structure is still available on c.facets,
but is deprecated, and will be removed in future versions. (#2313)

Our biggest release so far (55 tickets) with lots of new features and improvements. This release also saw a major new production deployment with the CKAN software powering http://data.gov.uk/ which had its public launch on Jan 21st!

Package Resource object (multiple download urls per package): each package
can have multiple ‘resources’ (urls) with each resource having additional
metadata such as format, description and hash (#88, #89, #229)