This article provides a step by step approach from ground up to devising your own malware classifier using machine learning fundamentals.

Naive Bayes Rule, statistics and ranking algos.

This is more like AV - Heuristics and other methods that dont have to depend exclusively on signatures, which are little more than file specific fingerprints based on byte pattern hashes or format anomalies that are boiled down to a detection checklist. However, this is still not a 100 percent perfected method though enougl calibration will certainly provide a workable and reliable enough engine. This discussion should encourage you to build your own variations and share their results