I’ve been seeing our full license revert to TRIAL on my dev machine for some time - I just presumed it was due to my frequent resets of the data directory, but we’ve just seen it on a pre-prod cluster, and are suspecting that it happens sometimes when sgadmin is run to update configuration.
Logs don’t show anything interesting - is there a package I can enable for a higher log level in log4j ?

Strange, we never had this issue before, so let’s try to debug. To clarify how the license handling works:

If you start Search Guard, and there is no valid license found in the Search Guard configuration index, we generate a trial license automatically. The start date would be the current date, and the license would run for 60 days.

If you update a valid license with sgadmin (by applying sg_config.yml which contains the license string) or upload the license via the API, the trial license in the SG index is replaced by the full one.

The API and sgadmin should prevent that a valid license is being overwritten with a trial license or an invalid license. Which means that once a license is applied, it should not be possible to overwrite it with trial.

I can only speculate at the moment, so I have a couple of questions. Your write:

“I just presumed it was due to my frequent resets of the data directory”

What do you mean by “resets of the data directory”? The SG configuration index is stored as any other index in the data directory of ES, so maybe this is the cause?

If your cluster with the license mismatch you described above is still up and running, could you please dump the current SG configuration with sgadmin? See “Backup and Restore” section in the sgadmin docs:

“resets of the data directory” referred only to my development cluster, where I would have been validating install/upgrade processes, ingesting bad data, bad templates, all sorts, so I would clean it out often, and the trial license didn’t worry me. That’s not the case in the qa/prod clusters where we’re now seeing the same issue.

dump the current SG configuration

Ha, for once I’m ahead of you - already dumped that out, there is no license data in the sg_config.yml file. I’d wondered whether somehow a trial license had found its way into our managed config yml, but it doesn’t look like it.

I find the issue/start date of the trial license to be very interesting - it doesn’t look like it was just generated anew, which suggests somehow a previous license has stuck around somewhere. I’m currently looking through logs for today and that start date.

Now that I find interesting. Yes, the trial license does not manifest itself in sg_config.yml, it’s generated and managed under the hood. However, date the trial license was first applied is indeed stored. That would explain the phenomenon at least partly:

a trial license was generated on 2020-03-13

a full license was applied afterward

then, somehow, magically, the full license disappears from the SG configuration index

then my guess is that the previous trial license kicks in again

Now the question is of course how a license suddenly disappears from the config index. If there is no license string in the dumped sg_config file, it means it is also not present in the index. This baffles me because I can’t think of any way to achieve this.

So maybe these are dumb questions, but: Did you notice any other configuration settings that are changed es well, or is everything running as expected apart from the license? Did you do any backup/restore of the SG index? Any other operation that might have an effect in the config index?

I can’t see anything unexpected in these files. The only anomaly I’ve seen recently (as covered on another thread here) is that auth backend config isn’t visible in the UI) which doesn’t seem related. Everything else is running normally.