Ad

Tuesday, June 26, 2018

Recurrent Neural Network (RNN) is useful for processing sequence data like sound, words, and time series data . RNN remembers a bit of the state from before. It can predict what will come next. Time series is good for heart disease over time, hormone level, blood sugar.

RNN weakness: sometimes gradients too close to 0 or too computationally large. It can also be bad at tracking long term memories - need to use LSTM instead, which has a forget gate, input gate, update layer, output layer

Hyperparameters in RNN

Hyperparameters are values that we need to set prior to applying an algorithm. Example: learning rate, mini batch size, epochs. There's no magical number. The optimal value depends on the task on hand.

Hyperparameter concepts: starting values, intuitions

Two main types of hyperparameters optimizer hyperparameters and model hyperparameters. Optimizer hyperparameters related to the optimization and training process more than the model itself. Learning rate, mini batch size, number of training iterations are optimizer hyperparameters. Model hyperparameters are variables that relate to the structure of a model. Examples include number of hidden units, number of layers, and model specific hyperparameters.

Yoshua Bengio: learning rate is the most important hyperparameter. Good starting point = 0.01. Also frequently seen: 0.1, 0.01, 0.001, 0.00001, 0.000001

Intuition for starting small (important): If our learning rate is perfect - the multiplier is the best, then in rare ideal scenario we will land at the optimal point. Any learning rate smaller than the perfect rate, will still converge, and would not overshot the optimal point. If learning rate is too large, will never converge (if it is more than twice the optimal rate for example. If it is close enough to the ideal rate, it may still converge. ). Intuition that is must start small. Udacity Deep Learning Nanodegree Part 5 RNN Hyperparameter No.3 Learning Rate gives a great visual illustration.

If learning rate is too small, may take too long to converge, wasting valuable computing resources.

Two frequently used models are Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN). Both are useful for recognizing structures in data.

Convolutional Neural Network (CNN) is useful for image recognition in medical imaging, diagnosis based on medical imaging, tumor diagnosis. Recurrent Neural Network is the less obvious one. It is useful for processing sequence data like sound, words, and time series data - understanding what will come next. Time series is good for heart disease over time, hormone level, blood sugar

Machine Learning for Radiology
The paper Implementing Machine Learning in Radiology Practice and Research (Kohli et al.) concluded that Machine Learning will assist radiologist rather than replacing their jobs. However, Geoff Hinton, a leader in Machine Learning, Deep Learning and Artificial Intelligence, thinks there's no question radiologist will be replaced https://www.youtube.com/watch?v=2HMPRXstSvQ.

CNN helps figure out what kernels, filters, and features to detect all the way backwards using a process called back propagation.

Back propagation is Neural Networks' way to update weights as in Gradient Descent. Back propagation starts from the correct answer, update the weights backwards layer by layer until the final classifications become more correct.

Don't have to see the entire shape. Should be able to recognize partial shapes, obstructed views. For example, a half hidden cow is still a cow.

Should be able to recognize different styles. Van Gough's portrait in post-impressionist style is still Van Gough. It does not change the identity or the classification of the image.

What to watch out for? Udacity tends to be experimental. Being in the first class may mean better price tag, but potentially extremely hard-to-follow incoherent content. I definitely had the issue with Machine Learning nanodegree when it first came out.

It does seem like one of those startup prototypes: standard format, basic content, with some industry professionals supporting the course including: Coinbase, factom, madhive.

Definitely worth it if your company pays the bill. Otherwise, wait and see better results.

# add and commit at the same time
# caveat does not add new files, still need to add it separately
git commit -a -m "comments here"

# undoing a commit
# move to the commit one before the current head (previous head)
# changes to go back to staging area
git reset --soft HEAD^
# or
git commit --amend -m "adding to the previous commit and also modify the message"
# or
#completely remove the previous commit
$git reset --hard HEAD^
# or completely remove the previous two commits
$git reset --hard HEAD^^

#sharing git
#dealing with remote repository
# git does not handle access control, it is usually is handled by GitHub or BitBucket

## always git pull from remote to ensure that there are no new changes
## if branch and master are at different timelines, i.e. not all new changes are committed
## in the branch that is about to be merged, then git will open an VI to view the changes
## git merge recursively and also create a node in the log about the merge

## merge commit
## if remote is ahead
## then have to pull
## which merge origin/master, the remote, with the local master
## then need to git push to add the new changes

Octal Notation:
01234567
10 11 12 13
one eight
one eight one one
one eight two one

r = 4 w = 2 x =1
first digit = u
2nd digit = g
3rd digit = o

rwx = 7 --- = 0
r-x = 5, -w- = 2

e.g. chmod 751 hello.txt

File Owner Ship
$sudo chown username filename.txt
will be prompted for password
$cat filename.txt
$nano filename.txt
but with nano not able to write, only view after changing owner
CHANGING GROUP NAME
$sudo chown username:groupname filename.txt
$su username
CAN NOW MODIFY because username is the same as the owner. Before when looking at ano with a the old owner which no longer have write permission
$nano filename.txt

$sudo
a super user command, run as root super user without having to know the password etc, sudo grants permission to other users, will requires own password not the root user password,

$!! runs the previous command
$sudo !!

Process
When running nano which is a program, creates a process, which is a running instance of the program. There can be multiple instances.
task manager for processes, resources consumed like memory or CPU, process has an ID, owner,
$top
question mark key brings up help function
e.g. sort the processes
capital F allows sorting
q to quit the program

Another program , will list out all the processes, represent running processes
$ps
(can see bash, and ps) ps is itself a program that creates a process, only in the current session
$ps aux
all process for user x, mean all processes for all users. it is not an interactive program. good for knowing what's running and what is the process ID. PID

Getting specific with piping, grep takes some input search for pattern only return lines with the pattern on it. This will also show up in its own ps search.
$ps aux | grep "top"