v1.8 Released

Improvements to search interface, ability to pass in parameters to extrators from the web interface, and more.

1.8.0 - 2019-11-06

Warning: This update adds a new permission for archiving files and adds it to the Admin role. Please make sure
to run Clowder with the MONGOUPDATE flag set to update the database.

Changed

/api/search endpoint now returns JSON objects describing each result rather than just the ID. This endpoint has three
new parameters - from, size, and page. The result JSON objects will also return pagination data such as next and
previous page if Elasticsearch plugin is enabled and these parameters are used.

S3ByteStorageService now uses AWS TransferManager for saving bytes - uploads larger than ~1GB should now save more reliably.

/api/search endpoint now returns JSON objects describing each result rather than just the ID.

Clean up docker build. Use new buildkit to speed up builds. Store version/branch/git as environment variables in
docker image so that they can be inspected at runtime with Docker.

Extractors are now in their own docker-compose file. Use Traefik for proxy. Use env file for setting options.

Utilize bulk get methods for resources widely across the application, including checking permissions for many resources
at once. Several instances where checks for resource existence were being done multiple times (e.g. in a method and then
in another method the first one calls) to reduce MongoDB query load. These bulk requests will also report any missing
IDs in the requested list so developers can handle those appropriately if needed.

Cleaned up pages to list and enable extractors. Added description of what the page does. Added links to extraction info
pages. Removed authors from table.

Added

Ability to pass runtime parameters to an extractor, with a UI form dynamically generated UI from extractor_info.json.
CATS-1019

Infinite scrolling on search return pages.

Trigger archival process automatically based on when a file was last viewed/downloaded and the size of the file.

Script to check if mongodb/rabbitmq is up and running, used by Kubernetes Helm chart.

Queuing system that allows services such as Elasticsearch and RabbitMQ to store requested actions in MongoDB
for handling asynchronously, allowing API calls to return as soon as the action is queued rather than waiting for
the action to complete.

New extractors monitor docker image to monitor extraction queues. Monitor app includes UI that shows selected
information about the extractors

New /api/thumbnails/:id endpoint to download a thumbnail image from ID found in search results.

New utility methods in services to retrieve multiple MongoDB resources in one query instead of iterating over a list.

Support for MongoDB 3.6 and below. This required the removal of aggregators which can result in
operations taking a little longer. This is needed to support Clowder as a Kubernetes Helm chart.
CATS-806

New Tree view as a tab in home page to navigate resources as a hiearchical tree (spaces, collections, datasets,
folders and files). The tree is lazily loaded using a new endpoint api/tree/getChildrenOfNode.

Fixed

Downloading metrics reports would fail due to timeout on large databases. Report CSVs are now streamed
to the client as they are generated instead of being generated on the server and sent at the end.

Social accounts would not properly be added to a space after accepting an email invite to join it.

Fixed bug where extractors monitor will not print error, but just return 0 if queue is not found.

Pagination controls are now vertically aligned and use unescaped ampersands.

Changing the page size on dataset, collection, space listings would not properly update elements visible on the page.
CATS-1030

Added a max of 100 status messages on the page listing all extractions. Before this trying to list all extractions in
the system was causing the JVM to run out of memory.
CATS-1032

Added padding to the top of the footer so that the superadmin notification does not cover buttons and the buttons
at the end of forms are not too close to the footer and difficult to see.