Filtering Out the Noise – Product Evals Using Trello
By: Rob Beason
One of the most important challenges facing all organizations is product selection. Most tools or solutions are significant investments, which no business can afford to get wrong. How do you determine if a solution is the right fit for your needs? How do you know if what you’re buying is just vaporware or another proclaimed silver bullet? While everyone struggles with these decisions constantly, too many organizations still don’t have a reliable process for evaluation. The specific criteria will differ across each product and business, but the approach should be consistent. As there is no reason to start from scratch, here is my process. These steps were selected to provide a comprehensive and even comparison between products.
This product review included a look at eight separate training and awareness products. The product names were removed to protect the innocent. The testing / review criteria of each product included three phases.
- Identification Phase (Can this address the problem?)
- Evaluation Phase (How effective is the product?)
- Selection Phase (What is Total Cost of Ownership?)
For each phase, develop a standardized criteria checklist to review the product against. The checklists are developed based on the business needs and mission requirements. This is the most critical aspect of the evaluation, as it provides guidance and ensures a consistent comparison.
The intent of this phase is to quickly review and eliminate bad options. Some solutions do not play well with your business size or mission requirements. Some products will require a large amount of resources. Any product that does not definitively address the problem should not be considered past this point.
This phase is intended to be the most in-depth phase of the product selection. This would include live demos, poking around the interface, and building the system in a proof of concept mode. For each checklist item, a rating will be given. If a product has a ‘cool feature’, that can be added to the checklist, or that specific solution can be given extra weight based on how it met the business need. The duration of this phase varies, but recommend setting a stop time, after all, a project without a suspense is just a wish.
This phase is intended to understand what the total cost of ownership would be for the organization. How much is annual maintenance? What’s the price per user per year? How well does the vendor support clients? How many hours per year will this solution require? How quickly does a vendor respond to critical vulnerabilities and issue updates?
- Keep It Super Simple: rating 1 to 5, 1 = run away 5 = awesome
- *Note: 1-5 scale could just as easily be 1-3, don’t get married to the numbers, just make sure to apply the same process equally.
- In most cases, all criteria will have equal weight. But, if one aspect is twice as important, add weight it by doubling its value.
- Then, add the totals and average it against the number of checklist items. The key here was to ensure the same general criteria is used for every solution evaluated.
At the end of the full evaluation, conduct a “Lessons Learned” exercise. Ask what went right, what went wrong, and how you can perform a better evaluation next time. This is one of the most important pieces of the process because you can use it to eliminate bad criteria, add more useful criteria, and flexibly mature the process.
As you move to your next product evaluation or tech selection phase, remember that the intent is to find the solution which BEST meets your business needs. There is no perfect solution, even though sales reps would all like to think their product can do it all. Be sure to methodically evaluate your options and stay consistent.
Rob Beason has demonstrated cyber security experience in multiple roles over the last 20 years. He started his career in the US Army supporting tactical communications for operational units on a global scale, has been a network security analyst at large oil and gas companies, and held positions as a security analyst and security operations manager for Idaho National Laboratory. He is an advocate for growing diversity in the cybersecurity industry and is actively engaged in volunteer organizations. His recent volunteer work includes helping the Idaho chapter of Infragard and Idaho Falls BSides.