Lance Norskog's Discussions

I've seen a lot of explanations of modern NN use that are all somewhat abstract. Tutorials usually just do one thing.I don't have a grasp of the exact mechanics of various uses. For example, an image…Continue

"What is the table format? Different table formats require different amounts of memory to create the output HDFS file. Text and Sequencefiles need very little memory.
Hive copies data from table2 to a local files on some of the servers, then copies…"

"The text search literature addresses this. A text search engine applies a scoring function to every document in the index, and chooses the document with the highest score. Text search indexes are all about pruning low-scoring documents early. "

""PCA destroys interpretability"
PCA is invertible, since it is just a linear translate+rotate matrix. You can apply PCA, fix the final N columns at their mean, and then invert the altered vectors back to the original subspace. This will…"

"I started DL not with TF/Keras/Pytorch but by directly coding all the arithmetic in Python, starting with this book:
https://makeyourownneuralnetwork.blogspot.com/
It really helped me understand the basics. I took his code, did some variations like…"

"Also- put up a production site. Just a web site where you upload cat pix, the app consults Google's API, then gives an answer. This is not data sci per se, but you're going to want to understand the quirks of putting Data Sci into…"

"An aspect you touch on in the first part is that an experiment is a sacred ritual. Someone who spends months worrying about her petri dishes full of fungus understands this deeply. Programmers are never taught about it.
"Design of…"

"Ummm... err... a fundament of Machine Learning is that 100% is impossible, and you expect to have a failure rate. In this use case, is a potential lawsuit the consequence of a failure? If so, rule engines are the right technology.
But that was not…"

"I think you want to do color constancy processing in HSV space instead of RGB space (as described in retinex).
Color constancy is a non-linear pass on color values which optimizes for similarity across different lighting situations. It is a hacky…"