I think we would also need a similar question for cases when the validation loss doesn't decrease (or does so only slightly). I tried to look for one, but I couldn't find it. Do we have one?

If the answer is "No", I'll write one with the same aim as Sycorax's. Now, to abuse the usual paraphrase from Tolstoy, "training issues are all alike; every generalization issue is particular in its own way", so it might be harder to give good answers to such a question, but I think that, if framed the right way, it could work. For example:

"My neural network doesn't reach the generalization error it should be able to reach, based on reputable sources or experiments with very similar NNs: which things should I check"?

For example, the first thing you need to check in this case is if you're overfitting the training set, and when (at which epoch) the training and validation losses start to diverge. There are a few questions where the OP only reports the accuracy on the validation set at a fixed number of epochs, which is pretty useless.

@amoeba I take it this means there's no such question, currently. Ok! Then I'll ask one later today. Hopefully it doesn't get closed as a dupe after 5 minutes ;-)
– DeltaIVSep 7 at 8:13

Having your question closed as a duplicate can be frustrating, but at least you would have found a way to ask your question that presumably nobody thought of before... when you have written your Q&A, can you please link there in an answer here?
– Stephan KolassaSep 7 at 12:52

2

@StephanKolassa I've written the Q. I plan to write the A by the end of the day (and I will share the link here) but if anyone wants to answer it before, be my guest.
– DeltaIVSep 7 at 13:14

@amoeba ah, but then Sycorax wrote the currently accepted answer to this question, where he correctly notes that several hours of careful crafting should go in the answer....also, you know: weekend ;-)
– DeltaIVSep 10 at 10:25

Sure I understand. But if possible, don't wait too long: you are rapidly losing chances to get on the Hot Network list which would increase the visibility of the thread (and the number of upvotes) by an order of magnitude. @Sycorax'es thread got there right after posting (I think he posted Q and A together or maybe in quick succession) and that brought 100s of upvotes.
– amoebaSep 10 at 21:43

@amoeba sorry, you're right! The answer ended up becoming so huge that I could never find the time complete it...I'll try to do better in the next days. Concerning the Hot Network list, I don't think the question ever made it to that list.
– DeltaIVSep 25 at 22:25

1

@amoeba at long last, I posted an answer! Looking forward to constructive criticism.
– DeltaIVSep 30 at 19:04

2 Answers
2

I'm not aware of a canonical thread on this topic. I think that making one is a good idea.

I have some general advice for anyone who is considering writing an answer. From my experience working on What should I do when my neural network doesn't learn?, I discovered that the trick is to be comprehensive and organized. Take the time to plan your answer. Use the formatting options to separate the post into distinct sections of related content. It will take time, but it's often the case that specific topics have specialized threads, so it's well-worth the time to dig through the site content to find relevant discussions.

It took me several hours to write my answer, and more time to provide incremental edits and revisions. It's well worth the time, since these threads will be viewed thousands of times or more.

I finally posted an answer here. It's less comprehensive that what I had in mind initially, but I could never find the time to complete the "unabridged edition", thus I went for a shorter and more focused answer.