install nautilus

I’ve tried installing a display manager, but instead I succeeded with directly starting the x server with xinit. related link

install xinit

$ sudo apt install xinit

create new xorg.conf

if it is a headless server, then the default xorg confs(located at /usr/share/X11/xorg.conf.d) is not going to work because it doesn’t have a real physical screen.

To be specific, I’ve tried launching the X server with lightdm.

$ sudo systemctl start lightdm

However, this does not work. We can see why it doesn’t work inside the xorg log file located in /var/log/Xorg.0.log

This log file loggs the procedure of launching Xorg. If the conf file is wrong, it will say that parsing conf file cause the error. If an error occured due to no screens, it will be said so inside this file. So it is important to keep checking this log file whenever X server does not seem to start properly.

Since the default confs do not work, we need to create a new one. reference link

$ sudo Xorg :0 -configure

a file named xorg.conf.new will be created in your current location.

If you have multiple GPU, then the very first section which defines ServerLayout will have multiple screens like below

The default setting will create a screen for each GPU and will attach them along side-by-side. But since this configuration will be using precious GPU resources, we can reduce to using just one. Remove all screen definition blocks except screen0. So this section will look like:

]]>https://kwagjj.wordpress.com/2018/09/15/installing-rmate-commands/feed/0mainuser000lazy summary adding in tensorflowhttps://kwagjj.wordpress.com/2018/09/09/lazy-summary-adding-in-tensorflow/
https://kwagjj.wordpress.com/2018/09/09/lazy-summary-adding-in-tensorflow/#respondSun, 09 Sep 2018 07:23:25 +0000http://kwagjj.wordpress.com/?p=2125Continue reading lazy summary adding in tensorflow→]]>When creating a training script in tensorflow, there rises the need to sometimes add summary protobufs later on in the same step. For example, lets say a training session is in play with a metric calculation step included. Periodically, I want to run a prediction with a validation/test data and record the metrics for these predictions along with the summary writer used to log the process of the training steps. In other words, a tensorboard image like the following is desired:

This capture of a sample tensorboard summary shows the progress of the training in “loss” and “metric” tab. There is another tab named “test” which logs the accuracy metric of the given test data. This only saves the value every 5 training steps and thus only three points are saved with the same scale of x-axis(number of steps).

When I was a tensorflow noob, I thought this was something hard to do. Thought that this would require a deeper knowledge of how summary writer works since I have to hack into it.

However after reading the docs more carefully and getting to know a little bit more on how the session,graph, saver, summary filewriters work with each other, this turned out to be something very easy to do.

The example that will be demonstrated is the one that gave out the tensorboard capture image above. I will use the inception_v3 model provided by default in the tensorflow package. The code is available in this gist link.

When building the model notice how I have two summary merging operations. The first one named train_summary_op will be pointing to loss_summary and train_accuracy_summary summary operations only. Don’t let the merge_all() function call fool you.

And after calling merge_all(), I then create another summary operation named test_accuracy_summary which is practically identical with train_accuracy_summary but only with a different tag name. And then I create a separate summary merge which contains the newly created test_summary_op summary operation.

This this strategy, I have separated the summary operations that I will need in training steps and in test/validation predictions.

See how these two merged summary operations work in different situations:

While looping through training steps, it will obtain summary protobuf by executing train_summary_op. The summary file writer will save the summary protobuf with the explicitly specified global_step set to the training step value.

In every 5 training steps, it will make a prediction with the test data and in this case test_summary_op will be executed in the graph. The resulting summary protobuf is added to the summary event file with the same writer. In other to retain the step number, notice that I have also specified the global_step parameter value to the current training step when calling add_summary.

This is the trick that I have used to record the interval validation/test prediction metric values along with the training. This trick can also be applied to save custom metrics that cannot be calculated inside the computational graph.

To demonstrate such situations, I will modify the example training code a little bit. The modified version is available here.

The changes made compared to the first example are:

Moved the accuracy calculating code outside of the TF compuational graph. In the new example code, it will be calculated in the main python thread using numpy and scikit-learn functions. Below are the related code lines:

After computing the metric(accuracy in this case) is calculated manually, we want it to be recorded in the tf summary event. To do this we need to create a summary protobuf, and in order to do this, we need to utilize the session. But we don’t need to flow through the entire model graph, but only a three operations:
placeholder that will receive the pre-calculated metric value

any operations that will convert the placeholder input to a normal tensor

Overall, some complicated metric calculations that are simply impossible(or extremely confusing and complicated) to do inside tensorflow computational graph can be simply moved out from the computational graph and execute it in a normal python thread. An example case where I faced such need was applying a custom NMS to the predicted bounding boxes. At the time I didn’t know that tensorflow offered its own NMS function and I wrote a python function that did this job. In order to use this python function on the predictions, it naturally lead me to calculate the metric outside of the computational graph.

I guess, if there are metric calculating can be done in the GPU, perhaps that may be more efficient. See if any metric/loss calculations that you thought needs to be customized are already supported by tensorflow. If not, and the calculations that you seek just seems to overwhelming to implement with matrice computations alone, then I advise you to try out this trick. There could be a loss in computation speed, but at least you can get the job done

]]>https://kwagjj.wordpress.com/2018/08/27/no-idea-why-gradient-is-not-applied/feed/0mainuser000when ubuntu apt failed to fetch/download info from a mirror repository(ex: kr.ubuntu.com)https://kwagjj.wordpress.com/2018/08/11/when-ubuntu-apt-failed-to-fetch-download-info-from-a-mirror-repositoryex-kr-ubuntu-com/
https://kwagjj.wordpress.com/2018/08/11/when-ubuntu-apt-failed-to-fetch-download-info-from-a-mirror-repositoryex-kr-ubuntu-com/#respondSat, 11 Aug 2018 02:47:50 +0000http://kwagjj.wordpress.com/?p=2118Continue reading when ubuntu apt failed to fetch/download info from a mirror repository(ex: kr.ubuntu.com)→]]>After upgrading to Ubuntu 18.04, I faced problems where the apt could not download deb files from the nearest ubuntu repository which was kr.ubuntu.com for me.

Strangely, when I access the repo from a web browser it seems to be just fine. But I guess for some unknown reason, my apt program was not able to.

The solution that I took was to force my apt to connect to the the repository in the US.

Referencing this post, here is what I did.

$ cd /etc/apt
$ sudo vi sources.list

In this file, switch all kr.ubuntu.com to us.ubuntu.com

After this, my $ sudo apt update command have downloaded the files without any problems.