Until now, we verify the kafkaprovider, by looking if the action in OpenWhisk has been executed with `activation list`. The problem with this call is, that the activation may not be in the list in time, because it only returns the result of an CouchDB view. If there is much load on the database, the view-computation may be behind and not return the activation.
As side effect, we use a trigger creation. The name of the trigger will be read from the kafka message. All other things (like credentials, ...) are already present in the action, that is invoked anyway.
To check if the trigger exists, we use the ID of the trigger. So no view is involved anymore.

Tweak producer settings to be more efficient
add retry behavior when the produce fails
shuffle broker list to distribute the initial connection load
Split produce into two phases
1. Establish and verify a connection with Message Hub - this can be retried
2. Produce the message - this can NOT be retried as it may result in duplicated messages

attempt to cache/reuse producer

This requires a slight change in the Python 3 action runtime that enables effective use of globals() across action invocations (under certain circumstances).

Limit number of cached connections

Arbitrarily set a limit of 10 cached connections
If you try to cache an 11th connection, arbitrarily remove one of the existing connections

* adjust for review feedback

* Make timeout values based on the invocation time remaining

Utilize __OW_DEADLINE to figure out when the action will be killed by the system
Use this value to determine appropriate timeout values for various aspects of the produce action
Reserve at least 10 seconds for the .send() call
Set a producer timeout that allows for three retries (minus the reserved 10 seconds)

Make use of a timeout value when constructing the changes feed. This ensures that if the connection to the DB is lost, an exception is thrown in a timely manner so that the changes feed can be restarted. Without this timeout set on the database client, the changes feed will simply hang when the connection is lost, and may not recover.
update to the latest python-cloudant package

This does not play well when using SSL connections to the DB. The fix is to make the DB client an instance property of the Database class, and also to ensure that all instances of Database are themselves instance properties of whatever class is using it.

This eliminates canary document deletes, which is good news for the changes feed as it won't have to process all the deletes when the container starts. This also avoids the problem of getting a 404 while trying to delete a canary that was already deleted by another instance.

automatically monitor and restore the health of the DB changes feed (#186)

* Monitor the DB changes feed by generating canary documents

Periodically add non-trigger (AKA "canary") documents into the DB. The changes feed loop checks for these canaries to verify that the changes feed is still working. If the time between canaries exceeds the limit, the changes feed is automatically restarted.

In cases where Cloudant is acting slow, it could happen that database migration is kicked off before Database initialization has completed. Moving this initialization into the main function prevents that from happening.

* If "trigger.suffix" is not provided, ensure test trigger name is unique

* Account for the fact that the test Kafka instance is shared

Because the test kafka instance/topic are shared, it is possible that the number of trigger activations includes messages that were produced by entities other than the currently-running test. Account for this by polling for a larger-than-expected number of trigger activations and then sifting through those to ensure they match expectations.

This change allows multiple instances of the feed provider to run simultaneously for the purpose of failover redundancy, and even a little load balancing.

This is accomplished by allowing the feed action to write triggers directly to the DB, without having to converse directly with the feed provider instances. The provider instances pick up trigger changes by using a CouchDB changes feed to detect new, modified, and deleted triggers.

To allow the feed action to talk directly to the DB, without exposing the DB credentials to all users, a new web action is introduced. This action can be invoked by any user, but cannot be inspected by anyone other than the action owner (typically a system admin account). The non-web feed action is still added to a shared package, allowing all users to create triggers from it, but this action only invokes the new web action over a REST call.

Even though multiple instances of the provider are running and, in fact, handling the same set of triggers, the use of kafka consumer groups ensures that each trigger is only fired once per produced message by way of the fact that Kafka guarantees that only one consumer in the group will get each produced message.

Batch messages until any of the following conditions are met:
- there are no more messages in the topic
- 2 seconds have passed since the last time firing
- the currently consumed message would exceed the allowed payload size

During /delete, if the trigger is (or is about to be) disabled, just delete it directly from the DB. Calling shutdown() doesn't work because the consumer thread is already gone and won't pick up the change in desired state.

* In recordTrigger() only set the status to active if it isn't already set

Allow for the case where a trigger that already has a status is being recorded.

Check if the health test trigger already exists, if it does, reuse it. If not, create it and do not delete it at the end of the test.

Added some additional logic to prevent cross-pollination of triggers being fired by other tests simultaneously running and producing messages to the same topic. Previously, this was prevented by having a trigger name that contained a unique timestamp. Now we poll for up to 5 activations that have occurred since just before the message was produced. Existing code already handled multiple activations, and rooted through them all to find the one that contains the expected message.