Thursday, July 24, 2014

Sometimes we just need to forward a port from one host, to another host. A few years back I built a java NIO-based port forwarder to learn about java NIO socket communication. The java source codes since has been lost and there is a different need for port forwarding, the first is that a server having failure connecting to services in another server, and I think that the server's IP might be translated to another IP while it were connecting to another network. So the port forwarding need to be able to log the IPs of the incoming connection. The second, is to temporarily circumvent strange network problems that prevent the first server connecting to another server.

Step 1 - Decide which incoming port to use

First we need to decide what port we are going to use to accept connections in the first host. Check first that the port is not already occupied in the host (try opening http://127.0.0.1:xx in your favorite browser where xx is the chosen port)

Step 2 - Open windows firewall for the port

For this step we need to go to windows firewall settings and allow connections to the chosen port.
In my Windows 8.1 laptop, the steps are :

Windows-S, type 'firewall'

click on the shown Windows Firewall icon

click on Advanced Settings (left menu)

click Inbound Rules (left tree)

click New Rule (right Actions menu)

choose Port, click Next

choose TCP, insert port number in specific local ports, click Next

choose Allow connection (don't change the default), click Next

check all Domain, Private, and Public boxes, click Next

type name and description, Finish

Step 3 - Enable IPv6 protocol in the network adapter

We need to enable IPv6 in the adapter, because Windows's proxy service needs IPv6 library even though we only forward IPv4 ports.

The steps:

Click triangle 'show hidden icons' in Windows Taskbar near the clock

Right click on connected the network icon

Click open Network and sharing center

Click on the active Connection where we want to enable the port forwarding

Click Properties

Ensure TCP/IP v6 checkbox were checked. If there is no TCP/IP v6 entry, click on Install, choose protocols, TCP/IP v6.

The log will be written to the specified path in the Log successful connections screen. To read the log, use Administrator command console, because the location is not accessible by normal user or Admin without privilege escalation.

Sunday, July 13, 2014

Background

Similar to the previous post, my yii-framework based PHP application need to access to oracle database tables. Yii requires PDO_OCI PHP extension in order to access oracle database. I will describe steps that I took to compile PDO_OCI extension from php package source SRPMS.

Preparation

In CentOS, we need to create /etc/yum.repos.d/source.repo because CentOS doesn't come with one :

[base-SRPMS-6.5]

name=CentOS-$releasever – Base SRPMS

baseurl=http://vault.centos.org/6.5/os/Source

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

priority=1

enabled=1

[updates-SRPMS-6.5]

name=CentOS-$releasever – Base SRPMS

baseurl=http://vault.centos.org/6.5/updates/Source

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

priority=1

enabled=1

We also need yum-utils and rpm-build packages

yum install yum-utils rpm-build

Then, download the source package file with yumdownloader :

[root@essdev ~]# yumdownloader --source php

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

* base: mirror.smartmedia.net.id

* extras: mirror.smartmedia.net.id

* updates: mirror.smartmedia.net.id

base-SRPMS-6.5 | 1.9 kB 00:00

base-SRPMS-6.5/primary_db | 672 kB 00:03

updates-SRPMS-6.5 | 2.9 kB 00:00

updates-SRPMS-6.5/primary_db | 104 kB 00:00

php-5.3.3-27.el6_5.src.rpm | 10 MB 00:49

I want to use a non-root user to do the compile, prepare the directories :

[esscuti@essdev ~]$ cd ~/src/rpm

[esscuti@essdev rpm]$ mkdir BUILD RPMS SOURCES SPECS SRPMS

[esscuti@essdev rpm]$ mkdir RPMS/{i386,i486,i586,i686,noarch,athlon}

Move or copy the src.rpm to the non root user.

[root@essdev ~]# mv php-5.3.3-27.el6_5.src.rpm /home/esscuti/

Do a test build

First we should be able to build the unchanged php source code.

Install the src rpm onto the src directories

[esscuti@essdev ~]$ rpm -ivh php-5.3.3-27.el6_5.src.rpm

1:php warning: user mockbuild does not exist - using root

warning: group mockbuild does not exist - using root

warning: user mockbuild does not exist - using root

warning: group mockbuild does not exist - using root

warning: user mockbuild does not exist - using root

...

As red hat said in Bug #206277, the mockbuild warnings are benign, ignore them.

Install build dependencies :

sudo yum-builddep php-5.3.3-27.el6_5.src.rpm

Do the build as the non-root user

[esscuti@essdev ~]$ rpmbuild -ba src/rpm/SPECS/php.spec

Oracle Instant Client Installation

1. We need oracle instant client for the OCI library, I have good experiences with instant client 10.2.0.4. You need to download Instant client 10.2.0.4 basic and sdk from OTN (oracle tech network).

Note that there are 4 parameters for with-pdo-oci that need to be specified. The first parameter makes an pdo_oci extension instead of statically compiling it. The second states that we will use oracle instant client driver. The third tells the location of the instant client files. The fourth explicitly states instant client version we have, because the autodetection doesn't work very well.

5. Rebuild using debuild, and copy the resulting so to the modules folder.

Thats as far I can recall. If something were inadvertently omitted, I would have to update this post.

Tips :

1. The linker would skip the instant client shared library (.so) if the architecture doesn't match. For example, it would skip the instant client .so if the downloaded instant client is for 64bit architecture and we are on 32bit linux.
2. Locale errors could be fixed by installing language-pack-de and executing locale-gen from root.

Tuesday, July 1, 2014

Background

Murphy's law describe that if something could break, it would break in the worst time possible. Or something like that. Anyway, more often than not, our software doesn't behave as it should. And I often get web apps that waiting endlessly for something. It made us curious what on earth cause the app to wait. In this case, the app is openshift origin console. Being Ruby based, means there supposedly a way to dump stack from running threads.

Technique

At first I tried to borrow the openshift ruby cartridge method of thread dump. Upon reverse engineering the cartridge (ok, I just snoop in some files such as this) I am surprised to find out that all that the ruby cartridge does is to send signal ABRT to the process which has the title prefix of Rack: . Trying to apply the same procedure to the running openshift-console process, and the result is a killed process and a confusion.

Another reference, the Phusion Passenger user's guide, tells me that Ruby & Python process that received an ABRT would print backtrace and then aborts the process. The fact that the backtrace were missing in any of the known log files after sending the signal ABRT made me skeptical about usability of this technique. The user's guide also states that signal QUIT could also be send to Ruby processes, and supposed to have the same result without killing the process in cold blood. But sending QUIT to openshift-console's Rack process also have no solid result either.

kill -s QUIT <pid>

Further lead

Knowing openshift-console's processes should contact openshift-broker's API, I starts sending QUIT to openshift-broker's Rack process. And this time, reviewing log files, I got several clues about the cause of the freezing openshift-console.

Summary

Unfortunately the post must end when the story is not quite finished yet. But the technique of sending QUIT to Rack process (or is it httpd process? forgot it) can shed some light about what region of ruby code currently executing right now.

This post will be about my experience installing OpenShift Origin in my company's servers.

Installation

I find the comprehensive deployment guide in OpenShift site (http://openshift.github.io/documentation/oo_deployment_guide_comprehensive.html) is very useful, but not without flaws. My OS is RHEL, but I think my experiences would also apply to CentOS systems.
The first glitch found was that the yum update went broken just after step 1.1 (repository configuration). The problem was a complex dependency between packages, in short it could be fixed after I do "yum erase libart_lgpl-devel". That package is not needed for correct system operation.
I noted that the mcollective installation (Chapter 5) as prerequisite doesn't mention that in RHEL we need different package than Fedora, that is

yum install -y ruby193-mcollective-client

Openshift installation liberally uses SCL (software collections) packages, in which a different root is used to install newer versions of certain software packages. For example, the ruby193 SCL is installed in (/opt/rh/ruby193/root/). The old version of ruby (1.8.7) is still installed in /usr/bin/ruby, and when we need to use ruby-1.9.3 we need to add the /opt/rh/ruby193/root/usr/bin path to PATH and LD_LIBRARY_PATH. It could be done by the command :
scl enable ruby193 bash
But the openshift installation does this globally by creating /etc/profile.d/scl193.sh, making the scl path available to the global profile.

For most of the deployment guide, from top to bottom is the correct sequence to apply. I said mostly because the missing ruby193-mcollective-client made my system without /opt/rh/ruby193/root/etc/mcollective/client.cfg to configure, and so I must do the configuration in later steps when the file would be available.

Complications arise because the newer version of v8 engine, previously packaged as ruby193-v8, now packaged as different scl altogether (v8314-v8 and or v8314-runtime). A new scl folder now required to be integrated to openshift applications, such as admin-console and developer console. If no adjustments were made, developer console shows blank in the browser and missing javascript runtime error shows in admin-console. In my system this is done by changing /var/www/openshift/console/script/console_ruby and /var/www/openshift/broker/script to include additional library path from the v8314 scl :

Network Troubles in Node

Several services will not start if your LAN device is not eth1. It is because most of the openshift shell scripts assume eth1 for network connectivity. Or it might occurred because I didn't specify EXTERNAL_ETH_DEV in /etc/openshift/node.conf. Anyway the cure is to set EXTERNAL_ETH_DEV in /etc/openshift/node.conf

Summary

Installing openshift origin in multiple host is a priceless experience. I get to know some of the mechanisms that openshift relied on.