Community

Post

XLR credential failure handling for XLD Deployment tasks.

For a request by Kevin Kocher

Problem Use Case

We've had a number of releases that get slowed down by RE/RM entering invalid credentials at the start and/or during the release that then fall into XLD tasks, most of the time many XLD tasks, and then fail. The credentials may be valid in XLD, just not valid for the current environment being deployed to. This can lead to a lot of confusion, for example, when the deploy tasks in one environment have already succeeded, but now the next environment in the pipeline fails with those same credentials.

Some attempt has already been made to try and minimize the deployment failures in XLR's XLD Deploy Task. It essentially revolves around the built-in XLR user input task, and the community plug-in "Get Latest Version"Once the input task is complete, we essentially hard-code a known application into the Get Latest Version task, pass the credentials the user just entered, and see if the task succeeds in getting the latest version of the hard-coded application.The idea here is that if the credentials don't allow the user access into XLD to run this task, it will fail and we can ask the user to re-enter them before moving onto the actual deploy tasks.

In practice, there have been a few problems with this approach.• 1. The validation is only on the credentials themselves. There's no validation that the credentials supplied will work with the XLD environment about to be deployed to. In practice we've seen good credentials get entered, the check passes, but then all the deployments fail because the creds supplied don't work in production.• 2. When the credentials fail, it's not very intuitive on how to re-enter them. Because the input task has already succeeded, and all the deploy tasks have already failed, really the only way to “try again” is to restart the phase. Power users of XLR can do this correctly, but not everyone involved with a reelase necessarily understands our instructions• 3. For some deployments the failures will occur in "everywhere at once" . This happens when individual applications are deployed in parallel. All of a sudden there are 40+ simultaneous failures to the release and many times each one has to have the proper creds entered on an individual basis. It's time consuming, and looks really bad from an end user's perspective.We’re trying to have discussions about what a sound approach would be that is both secure and “easy to use”. Not only for successful cred validation, but also for easy retries.

To circumvent all that retry we’re also considering defining a single user in XLD with “deploy everywhere” access, but we don’t really know how to secure any XLR task that might use those creds.Someone with edit access to the release as a whole could go into the task to modify the app, env, or dupe/change the task type, and be malicious (accidentally or on purpose). Maybe if there was a way to control access down to the task level?

Would you have any suggestions or other solutions in the works or implemented that might help us reduce these types of failures?

Solution Approach

We can create a custom tasks call it validate deployment users.. put it as the first task in releases.That task would search for all the XLD tasks and makes a query each with they deployment app/env/creds and prints a report of which one would fail. It can pass if everything works or fail there printing the report about the failing task. The folks can then correct those tasks, retry this validate task and if it passes, they should be all set

Prototype ( works on 6.1 )

Using the above approach, i wrote a script that can be used inside a script task as the very first task in your release. the task would look out for all XL Deploy tasks and try to capture the information from them, make a call to verify the deploy#initial permission for the credentials on that environment and report it back in a summary. This ofcourse can be wrapped around nicely in a plugin and then used. Also would require a little bit of change to make it compatible with XLR 6.0 and below.

9 comments

We went down the "single user in Production" route to make things simpler.

Our current methodology is:

1. Create a Service Account that has "just enough" privileges to achieve your aims.

2. Create a "Production" Folder and a "Test" Folder:

- Test: No Credentials stored, users can edit releases and templates

- Production: Only Production Credentials stored. No "user" has edit rights

3. Design and Build in the Test Folder. Users use their own credentials and rights, so Authorisation is dealt with elsewhere.

4. Once a Template is finalised (signed off), Admins move it into Production and cache credentials.

5. If the Template needs changing, it must go through another round of test/sign off.

6. Certain Admins have edit rights on Releases (not Templates) via an external Audited Password Safe to use during issues arising during the run.

Removing the power to change the template forces the designers to implement checks, tests etc as well as ensuring it receives backing from all stakeholders.

Once it's in Production, we use our Password safe to ensure that only authorised users can start the Release and Gate tasks to ensure sufficient agreement before anything goes through. Some teams still prefer to not cache credentials for sensitive applications or for server access but for XLDeploy Actions it's pretty much standardised.

The difference between 6.0 and 6.1 stems from the fact that in 6.1 we converted the XL Deploy task from a built-in task to a plugin. We tried to keep it as transparent as possible, but when using it in scripts the new structure shines through.

Hi Amit,Thank you for running with this! It sounds very promising.So just to understand the high level flow. The release would likely be created in a way that all necessary deploy information has been entered prior to the script task executing. The script would then run, and then if there was a failure due to say, bad creds (stored as variables on the creation of the release), the release owner could then go to the variables page to update them, and then retry the current failed script task? Is that accurate?

I like this "dry run" approach because it would allow us to shake out errors ahead of actual deployments in a safe and reliable manner.

Hi Kevin,
Glad you like the approach. What you are saying would precisely be the way it should proceed. I haven't tested this using variables but i was putting the values directly and so it worked. It might need some small changes. However if you noticed that i am using it by putting directly into a script task whereas for you it might be more convenient to be wrapped into a custom task since that restricts any other user fiddling with the script or even a typo causing it to fail.
Regards
Amit

I further wrapped that the script into a type such that it can be used as a task. i am trying to keep it generalized so it can be used for XLR 6.1 and older versions . with the old versions it needs to make a request back to XLR server for fetching the XLD server references. there's some minor caveats but they can be worked out.. try it.. you can use either the jar file and drop it in plugins or use the synthetic and script directly under your ext.

I was intrigued by your problem so got interested to write something that might work.This is more of a dedicated services work but as long as this script works for you, you should be fine.. Test it thoroughly as there are some small caveats

I would suggest approaching for some services if you further want to customize this code peice. I might write a blog about it eventually but on time crunch these days.

Just an update. We were able to evaluate the plugin 'as-is' using a sanbox env, and the overall response was very positive so thank you again for doing this!

Functionally it's doing exactly what we want to do. Take the deploy env into account and give us a chance to fail before failing the tasks that would have failed.

We found one main issue. I realize this code is something we can use as a base and run with ourselves, but just so you're aware.

The code doesn't seem to handle passwords with special characters (like $ for example). Worse, the failure ends up showing the attempted password in plain text:Exception during execution: KeyError: 'yourplaintextpa$$word' in <script> at line number 86

This also gets logged by the server so it actually ends up in 2 places. server logs and an xlr task.

Thanks Kevin, I am glad you are able to use it. I know it had some kinks and i tried to make it compatible to both old and new versions but it has some missing pieces to make it production ready. That would be a great if you guys want to play with it and further enhance it. It is a nice one that you can publish formally on community if you'd like. It would be nice if you can maybe approach services for a quick one to help you formalize on this.