Experimental evaluation with cross-validation


It is extremely easy to make methodological mistakes when evaluating some machine learning system and comparing it with others.

Especially when using cross-validation ! A must-read paper about this topic is:

On Over-fitting in model selection and subsequent selection bias in performance evaluation

So even though you are short of time, whenever you are running model evaluation, please always take the time to double-check that your evaluation methodology is flawless, especially with cross-validation experiments, where you must always tune your hyper-parameters with a nested cross-validations procedure.