Since we are picking nits, it is a bit confusingly worded. I would suggest that "not a small nor trivial problem". To me, at least, the "nor" assures that the "not" applies to both "small" and "trivial".
The author could say, however, that the use of "or" implies that the "not" does not apply to "nontrivial" and that the original is correct.
But, IANAET (I am not an English teacher), so what do I know.