queuekit

Queuekit is a plugin for grails which uses TaskExecutor with Spring Events for grails 2 and for grails 3 using

default Reactor events to manage concurrent submitted reports.You can use it for your own custom reporting needs or
simply combine it with grails export plugin to queue export requests.

At work we face a scenario where reports and core application are integrated and even if separated they would

still hit the same database.

The reports are typically rendered through response as a file stream and saved to user's desktop.

At some periods of the day the application runs slow, we think due to people running reports concurrently.

Without going into further complexity of database, application and reporting system. A cure would be to limit the amount
of concurrent reports that can run and sort reports based on a priority system allowing (HIGHER: Speedier ones through
over LOWER: Calculations would suggest it will take long per report basis)

3.Information

=====

Provides a variety of methods managing your queue:

ArrayBlockingQueue - This is the most basic and provided are database functionality to manage the queue for you
ArrayBlockingQueue has no mechanism to manage incoming threads, it will take on as much as available and beyond that
reject them. Additional steps have been added to capture / re-queue those that exhaust the queue and to manually check
the queue upon completion of last task.

LinkedBlockingQueue - This manages the queue automatically
LinkedBlockingQueue is the next phase up, Since it manages the queue for you. If you have 3 maximum threads and fire 5.
2 will wait until the first 2 are done and then their picked up. Queue is processed and limited to items as they arrive.

PriorityBlockingQueue - This manages the queue automatically and also attempts to run concurrent requests with a priority.
PriorityBlockingQueue by default provides a mechanism to manage queue items according to each items priorities.

EnhancedPriorityBlockingQueue - This manages the queue automatically and also attempts to run concurrent requests
with a priority. PriorityBlockingQueue by default provides a mechanism to manage queue items according to each items priorities.
It also binds a new thread task to an inner thread. This means you can also cancel a live running thread.
When a cancel request is issued. The main running thread also kills off the inner thread that is the running task.
This required further concurrentHashMaps to track / remove elements.

With both PriorityBlocking and EnhancedPriorityBlocking which was really my own additional work around PriorityBlocking,

The priority itself is defined by you in your Config.groovy depending on the name given to it.
There is also an additional hook that you can add to all/some of your reports that will go off and look at custom
parameters provided by the report and decide if it should lower/higher the default config priority.
An example for this is also provided. The policy as to how it decides is entirely based on what works for you.
You could use the example to expand on your business logic.

Must outline the topics covered in point 4 EnhancedPriority are all very experimental,

it's the end ambition/desire to achieve all that without any side effects. I think you may find odd behaviour.
Your input / enhancements are most welcome.

If you have 3 threads and 6 LOW Priority reports launched

If after (reportThreads - preserveThreads) = 3 - 1 = 2 (So only 2 would run at most at any time of LOW Priority)

This means all those below Priority.MEDIUM will have a spare slot to run within. In short an fast lane left open always
for those below medium and 2 slow lanes. You should be able to configure 6 report

With admin rights you can:

> Change a priority of a queued report from the main listing screen.

But beyond that when a report is queued you can click on report item and choose change priority. If it was LOW and set
by Config or override hook you now as the human can set it to be a higher priority which will take effect when next
thread becomes available.

This means you can on the fly increase or decrease and change preservePriority group from the main report screen.
The changes are only for runtime. Meaning upon restart the defaults from Config.groovy or if not defined what
plugin thinks is best is used.

> Shutdown ThreadExecutor

No idea why you want to do this unless you are testing in worst case scenario with intentions of testing
useEmergencyExecutor=false or manualDownloadEnabled=true

> Control maximum running time before killing a running task and canelling it.

configure killLongRunningTasks=300 where 300 = 5 minutes in seconds. Provide the time in seconds you wish to wait
before the taskExecutor sechedule is killed off. This only works for EnhancedPriorityBlockingExecutor and when it
attempts to pickup the next thread, it checks running task of existing threads.
If any match your set limit and beyond they are killed and set to status Cancelled

With Configuration you can:

> Configure report time highlight

In this example if report takes:

over 01:10:02 (1 hour 10 minutes and 2 seconds) it will highlight in blue
over 00:01:05 (1 minute and 5 seconds) it will highlight in orange
over 00:00:12 (12 seconds) it will highlight in html colour code: #FFFFAA

> Configure a backup Executor for when / if main Executor is down

When tasks go into rejection it is typically due to Executor having issues. This is all true for all Executor
types provides besides ArrayBlocking.
So this feature is available for all besides ArrayBlocking. Since ArrayBlocking has no queue mechanism and is managed
by plugin DB lookups. It automatically puts a new task in rejection if over limit running. Therefore we capture those
and re-queue them in this case. For all other cases if you enable
useEmergencyExecutor=true This will tell the plugin to fall over to a single executor and launch a schedule of the
user request. This is all transparent to the end user and happens in the back-end. It will throw errors in the log.
All the rules of priorities goes out of the window and there are no limitations as many requests made is as many
threads launched. It keeps business flowing whilst a fix / resolution is found I guess.

If main Executor is shutdown - due to how it is wired - (behaving like a service) it needs an application restart
for it to reset. Ideas welcome, some comments in EnhancedPriorityBlockingExecutor in regards to this.

> Configure a backup of a backup or disable backup Executor and fall back to manualDownload

Like above if tasks go into rejection, if you have configured backup executor and even that appears to be shutdown -
which would be strange since the backup is a single executor launched per request. Either way, you can also set this
to true manualDownloadEnabled=true in your Config.groovy.

This again is transparent to the end user, but if all of above has failed it will actually launch a new single
runnable task to execute the task, the report is treated like all of above, it is captured in report listing, shows
that it is running and completed and is also timed. But has totally bypassed all of the threadExecutors and just
launched as a thread within application.

This also behaves like all of above meaning the end user will get a prompt report has been triggered and if trained to
do all ove above they would go to the normal listing screen and wait for it to complete.

If you wanted to make this fail over behave like a real download, you could refer to ReportDemoControllerindex
action which has the following:
(It makes the user wait on the screen whilst the manual thread goes through the process of being created. When it has a
file, it redirects to download page like they would have if they had clicked on a live report download action.

4. Additional simplification / explanation

This plugin is from a concept I put together for work and will convert the process of user report interaction from one of :

Click on a report - or define report criteria and click download

If the process is to then go off and get data produce a CSV,XLS,TSV,DOC,PDF of some form and you are using your controller request mechanism
to deliver file through a stream.

If above describes your scenario then as you are probably well aware, as database/user-base grows and more reports are
requested. Specially concurrent requests by many users can have an impact on your application performance.

This plugin will be able to change that process and limit to concurrent threads as well as provide a queueing / user
report interface. Since the jobs are converted to background tasks the files are produced when system has completed
and user has to check another interface rather than on demand file generated on the fly as they clicked save/download.

The process to convert your existing reports should be really kept to a minimal so long as Controller was used to
generate file from some result set that came back from some service. The only minor change is where you
defined request type and out and filename. These segments can be stripped out and rest will be near
enough what you had :

Examples

Binding queuekit with grails export plugin

Most importantly pay attention to bufferedWriterTypes configuration ensure you have removed CSV and csv from it. If you
are going to use this plugin for working with export plugin. The export plugin uses a different way to export csv compared
to how you would normally through a controller as described further down under manual reports.

You can use this plugin in conjunction with the export plugin to essentially change the mechanism from files produced
as requested to user requests for export plugin being quueed through queuekit.

Please feel free to browse through the grails 3 demo site which has all of this in place, I will show the more
advanced version since it is probably most feasible.

You have installed/been using export plugin, you have configured the required addition in the controller to send
report to exportService.

I installed the plugin, created a domain class generated controllers and views and then amended the call to
exportService to :
queueReportService.buildReport(reportName,userId , locale, params)

That's it, the user reports will now be queued through the queuekit plugin,
You can see export feature is called in the ExportPluginAdvancedReportingService.
The code in the controller can be copied from controller to controller.
Just pay attention to: (Ensure you are passing in correct domainClass that you will use in the sharedExport service.

/**
* ! -- IMPORTANT
* In order to let this dynamic exportPluginAdvancedReportingService pickup the correct domainClass
* We must send an additional params as part of reports calls and bind in the actual
* domainClass we would be listing
* just like shown here
*
*/
params.domainClass=TestAddress.class

Manual Reports using plugin with your own methods of producing reports

Apache-poi XLS files - manual report

Check out grails queuekit demo site for grails 3. Follow the example to see how I got it to work
All very similar to instructions below besides that it is using custom libraries to produce the output. (different file types to standard csv/tsv described below)

Then you are half way there, in principal the same thing would be put in to your service that would be in the action
report. The plugin handles out so there will be no need to define response or out variables. So all of above would become:

Other useful information

queuekitUserService def userId = queuekitUserService.currentuser

This is a userService that exists within this plugin, you should override this as per example site and feed in your
real user/userId/userLocale/permission values in from your own site.

reportName String reportName = 'tsvExample1'

This is really as important as it gets, ensure you use proper class naming convention so no +_&*^!£$%^ characters no
space etc just normal alphabet as if you were naming a domain class.
Create a new service called

Binding application security with the plugin

Under the grails 3 demo site, spring security got installed a new
service called MyUserService which extends QueuekitUserService and overrides the default actions of the plugin to return
userId if user is a super user and so forth.

The service then takes over QueuekitUserService the test site's grails-app/init/test.queuekit3/Application.groovy

//Inject the service
def queuekitExecutorBaseService
//Run this
queuekitExecutorBaseService.rescheduleRequeue()
//Also ensure you have enabled in your Config.groovy/application.groovy
queuekit.checkQueueOnStart=true