Artificial Reality | The PatternEx Blog

Success Criteria for PoCs

As you are already aware, undertaking a proof of concept (PoC) is not a trivial task. When executing an AI based security evaluation, the process can change compared to traditional IT projects. To get value out of a PoC, you / your team need to put effort into it. So, what are the steps to follow to get the most out of a PoC - and ensure that the product or service being evaluated will actually meet your business needs?

First, have you actually articulated your requirements - in writing? Have you vetted those requirements with your team? Have you then vetted those requirements with your manager?

Once requirements definition has been completed (and approved), then you can begin to informally evaluate vendors for how well they meet your requirements. Your PoC might be a single vendor evaluation, or it might be a bake-off between multiple vendors. Once your vendor candidate(s) have been selected, then your next step is to discuss specific use cases to be evaluated in the PoC.

Along with requirements and use cases, you must also agree on resources (hardware, software, personnel, access, etc.) needed to conduct the PoC. Don’t forget about data sources for the PoC, including specific data elements needed. With that accomplished, you will need to agree on success criteria. How are you actually going to evaluate the vendor(s) in a fair and objective way? For example, what is the time to detection? Does your organization have an existing solution that you can compare results against (i.e., A/B testing)? Should your organization do a POC only with existing data sources in your SIEM? Or, should you explore other data sources / use cases that were too computationally expensive previously, or not being processed by your SIEM?

Assuming that the PoC goes well, you’re not done yet. How will be results of the PoC be delivered to you? Face-to-face meeting or by phone? Presentation, or with additional technical results? What about customer references from the vendor? Hopefully, at least one of those customer references is in the same market segment as your organization, as well as the same size, personnel-wise.

Depending on what business problem / use cases you are attempting to solve, there may very well be other aspects to the PoC to work out in advance. For PatternEx PoCs, additional considerations include:

Data types: which data sources are available in your environment?

Ingestion mode: is the data going to be available in a real-time or batch mode?

Data sizing: what is the volume of logs per day (in GB) for the POC?

Infrastructure: are you planning on running the POC on VMs on-premise, or in the cloud?

Internet access: will the system have internet access (package download)?

So, want to conduct a PoC with us? Let us know. We would love to find out how PatternEx’s Virtual Analyst Platform can help your organization to improve your detection capabilities.