How best to avoid asset duplication? (VM & Agent)

I was hoping to hear how some of you avoid ending up with duplicate assets (cloud agent & ip asset imported on scheduled inventory/vulnerability scan)? Unfortunately I need to scan the same range as the cloud agents sit on to ensure nothing has been missed, which results in the duplication at the asset levels. (I know we can merge the asset but this is only at the reporting level and not asset inventory) - we integrate with Kenna so the Asset Inventory is the gold source.

I've managed to speed the scans up by creating an light weight Inventory scan of the subnet that allocated a specific Site tag, and then a Vulnerability scan that scans for Site tags but that excludes the "Cloud Agent" tag.

The problem is that even if I purge the IP assets that are duplicates of the Cloud Agents, the next time the Inventory scan runs it will re-add the asset.

I'm trying to automate the process as to not need to manually tag things and would like to avoid the API (unless absolutely necessary) Anyone any ideas?

We have gone through the same thing with Qualys. Three things that we did that helped us was:

Authenticated scanning

Agentless tracking

For our agent-installed assets for servers we created a network scanning job that defaults the agent as the authority. Basically it scans for everything the agent doesn't track (port-related)

Before this, we were scanning full network scan and agents were reporting in. This caused a lot of fighting between network scans and agent scans. We drank the Qualys Kool-Aid and created this scan and it has been working pretty good since then. It has definitely cut down the QID flapping (scan fighting each other).

I sent some information to Qualys support to take a look at and it would appear that our Agentless Scans (from the Asset we provided) were not able query the Host ID value on these machines. This would be the reason we are getting the duplication for our case as if the Host ID cannot be retrieved then the asset would be unable to merge with the Agent results.

Example:

QID 45180

Threat / Description:

Report errors related to accessing the Qualys Host ID from a target host. The host ID is used by the agentless tracking feature, which allows customers to track hosts by a unique host ID, instead of relying on the IP address, DNS name or NetBIOS name to identify a host.

Impact:

Customers won't be able to troubleshoot issues related to accessing the host ID from a target host.

Solution:

The results contains the error codes and the description of the error.

Hi peterlyttle. Yes, one more configuration option needs to be enabled. Please check to make sure Unified View is turned on. This is under Users -> Set Up -> Cloud Agent Set Up. Details at this link: How to Merge Agent Data

there is something called unified view for agents and non-agent combined environments. but we are dealing with a primarily non-agent based duplication issue on our side. asset view and even in host detection table we pull from with API does not properly dedup/prevent duplicated assets. we use the long character string QG host ID and still see duplicates, we changed a large portion of our subnets to IP tracked based on feedback from Qualys and still have the issue. In fact we have multiples of the same asset, reported with the same QG host ID scanned at different times, reporting different vulnerability values when the asset host detection info is called. I think there is something not aligning with Qualys' backend mapping/tables etc.

This duplication is often caused by one of three things. Unauthenticated scans, the account used for an authenticated scan having insufficient privileges to read the appropriate asset ID's in the registry from agentless tracking, or an ID got saved in the cloning process when standing up a machine and never purged properly. Can you send me a bit more information about what you're seeing? We'll get a ticket open for you Adam so we can get the right people on the case.

Our number of duplicates has decreased over the past few years, but we still see this issue on a few assets. Especially ones with multiple IP addresses even when they have the Qualys Cloud Agent on them. I think part of the problem has to do with each IP address does not necessarily have the same "privileges" to the asset. So, everything works correctly when scanning IP address X, but not on IP address Y.

If you have assets with multiple IPs and the Qualys Cloud Agent on them, do you also scan with an appliances? If so, do you also have an authentication record in Qualys for the asset? Does that authentication record have all of the IP addresses for that asset in the authentication record? If not, you might try it for that one asset and see if it resolves the issue for that one asset.

Other thought, if you use the disposable agent when performing scans with the appliance, does the agent fail? Check for QID 90918 - Dissolvable Agent installation failed. This can happen for a few reasons. I typically see a lack of disk space.

Yea I've seen that same thing with L3 switches, where the interface address has no authentication but the management IP address does. That's something that is inconvenient but we can live with (though would be great if Qualys could resolve based on serial number or something). Though in the OP the devices in question only have 1 ipv4 record.

The VM scans are authenticated and tracked via IP - I don't see any auth failures or disk space limitations.

We have gone through the same thing with Qualys. Three things that we did that helped us was:

Authenticated scanning

Agentless tracking

For our agent-installed assets for servers we created a network scanning job that defaults the agent as the authority. Basically it scans for everything the agent doesn't track (port-related)

Before this, we were scanning full network scan and agents were reporting in. This caused a lot of fighting between network scans and agent scans. We drank the Qualys Kool-Aid and created this scan and it has been working pretty good since then. It has definitely cut down the QID flapping (scan fighting each other).