Dec 3, 2013

I found these functions very useful often because it doesn't use any run-time computation cost. Calculations are done at compile time and intermediate values are not stored into the binary file.
It may increase compile time a lot if you try to get result with big numbers and only constant values can be used; no variables are allowed.

Dec 1, 2013

I made a Lexer/Parser/Application of C-preprocessor with Antlr and C#. I like to share my work with anybody who may have interest in doing it by himself or herself. *PS: Antlr is a parser generator.

Of course, there will be many or better ways to do the same thing than the way I did. Here is a summary of how I did.

Lexer A and Parser A take a C/C++ source text string and parse macro keywords such as #include, #define, #if, #ifdef, #ifndef and so on.

Application A stores #define keywords parsed from Parser A; there are two types of macro defines: one that takes arguments and another that doesn't take arguments.

Whenever #if or #elif are encountered, the macro keywords in the condition statement are resolved with the stored define macros. This process is done by Lexer A, Parser B; Lexer A is reused. This step is repeated until there is nothing to be replaced. Once all of macro keywords are resolved, arithmetic calculation is performed with Lexer B, Parser C and Application B.

In the line1, a new macro keyword, "VariableType", will be stored by ApplicationA.

In the line2, a new macro keyword, "FunctionType", will be stored by input argument information.

In the line4, the macro, "FunctionType( (2*3), 2 )", is replaced by the definition and it will result in a new string, "(2*3) - VariableType * 2". The replacing step must be done after proper parser and that's why I needed a second Parser B for the macro resolving. For example, without Parser B, simple string replacement will cause errors in cases like "FunctionType( FunctionType( 1, 3 ), 3 )".

Since a macro definition can include another macro define keywords, one time macro resolving is not enough so the step had to be repeated until there is no macro keyword left.

Once all of macro keywords are resolved, the calculation must be performed based on operator precedence. That's where I needed 3rd Parser C and second Lexer B. For the example, the parser will parse a statement, "(2*3) - 4 * 2 + 1 * 2". This is a relatively simple calculator, which is Application B.

Based on the calculation result, one of statement blocks, which is line 5 or line 7, are ignored and another statement is processed. In the example, the calculation result is 0 and line7 will be processed.

I can share the Antlr grammars here but application source codes are too long to be posted; I wrote C# applications.

I am hoping that this information can be helpful for anybody who is planning to make their own version of C preprocessor or any kind of preprocessors.

*PS: I read one IBM guy suggested that all of C preprocessing should be done by Lexer only. I don't see if it is a better way or not because as I described, C preprocessor also need parsing steps several times.

Nov 23, 2013

I made a little complex but useful batch function for comparing the modified timestamp of two files. It is DOS batch command. BTW there is no command that does this on batch command without help of any external utilities.

The way it works is get DIR result and parse date/time info. I convert year/month/day/hour/min into one big comparable integer value. Two timestamp values are subtracted and if the result is negative, it means the second is bigger value. For example:

SET /A C=%A% - %B%IF "%C%" == "-" ECHO B is bigger

Here it goes:

:COMPARE_TIMESTAMP_GTREM %~1 is first file name that needs to be compared toREM %~2 is second file name that needs to be compared toREM %~3 is return variable name; if "%~1 > %~2" is true in terms of timestamp, it returns 1; otherwise 0.IF NOT EXIST %~1 GOTO :EOFIF NOT EXIST %~2 GOTO :EOFIF "%~3" == "" GOTO :EOF

With HandBrake, I couldn't find a way to convert only audio but not video. What I wanted was not touching the video for quality reason and convert only the audio to AAC or MP3.

I found several people posted how to do that for MKV container format. From those scripts I made my own script that converts audio tracks. I don't know how to manipulate AVI or other container formats yet but most of cases MKV seems to be enough.

This script is faster than transcoding both video and audio because this converts only audio not video. And since it doesn't touch the video, the video quality wouldn't be affected as well.

I believe as long as you use Raspberry Pi, you wouldn't need to transcode video. But if you are using iDevices, you may need to transcode audio and video both.

As you can see it generates temporary files on the way. You will need to manually remove them or you will need to modify the script by yourself. I just don't like any scripts to remove anything for safety reasons.

Jul 14, 2013

I explained how to store video data from Foscam IP Camera on Raspberry Pi. Now I am going to show you how to wire them all together.

As I explained, Foscam IP Camera, FI8910W, has motion detection feature in it. When any motion is detected, it will either send an email or upload a few still images to FTP server. A problem is that it doesn't store video data. That's what I am trying to do in this article.

The trick is that on the camera side it will do the motion detection and uploading files to FTP. When FTP connection is requested, it actually interact with a fake ftp server and the fake ftp server will trigger the video recording with a script that I explained.

Here is a big picture of how whole thing works.

It opens a fake ftp port and wait for FTP request.

Camera request FTP login.

Start recording video stream

Wait for another FTP request to identify whether or not there is motion going on still.

If FTP access is request within a given time, it keeps recording the video stream.

If FTP access is not requested within a given time, it stops recording.

I placed it here: /home/pi/script/foscam_fake_ftp.sh
But you can place it wherever you want.

In the script, a special character, "^M", is not two characters but one character. In VI, you can type in the character by pressing "Ctrl+V Ctrl+M". The special character was entailed when the camera send text message so I had to remove them in the script.

A difference compare to the previous Windows batch file version is that the ID and Password information given to the fake ftp server will be used as a login information to access the video data. This time, we don't care whether it is Passive or not.

I spent whole day to implement this code. I think there will be many people who would like to use Raspberry Pi as a IP Camera recorder so I am sharing it here.

This is a shell script that uses GStream 1.0 library, which is not an official Wheezy dist yet. The reason why I am using it is that it was only one way to use hardware accelerated H.264 encoder. Currently only gstream-0.10 is officially available not 1.0 yet.

If you are not comfortable using unverified package, you can also use VLC without hardware acceleration. I haven't tested VLC way thoroughly but it shouldn't be too hard for anybody to use it.

Also my script uses "wget" to retrieve some information from Foscam IP Camera such as "alias name" and "alarm_upload_interval". If you don't want to install wget and you know those values for sure, you can simply modify the script and hard-code it.

The last line is the core of the script. The logic is that it retrieve ASF video streaming data from Foscam IP Camera with "souphttpsrc". We need to decode the video with "decodebin". Then it becomes "raw video". The raw video is piped into "videoconvert". Now it is passed to the hardware accelerated H.264 encoder, "omxh264enc". I don't know about "h264parse" but without it, it didn't go through so you need the step as well. Then the encoded H.264 video is stored as MKV with "matroskamux" plug-in. Finally the file name for the output data is specified with "filesink" plug-in.

I found that two steps, souphttpsrc and decodebin, can be merged with "uridecodebin". It seems that uridecodebin can handle "buffering" feature and probably it would work better in different cases.

Another thing I want to mention is that stderr of gst-launch-1.0 should be redirected to /dev/null. Otherwise, it will make the script process "defunct". I think a child process of gst-launch-1.0 is holding the stderr of the parents' and it causes defunct processors when those parents are dead.

Jun 17, 2013

Today I remembered something that I was wanted to do before I come to America, which is 6 years ago. Web banner.

It was super easy to apply for Google AdSense; it took only 5 minutes to apply. And a few hours later, I got an email saying they rejected my application due to copyright material on my site.

First of all I was disappointed. I thought I can motivate my self for posting more quality articles on my blog by making some pennies. My excitement was changed to a big disappointment.

Second of all, I felt offended. I checked all the articles on my blog three times. Most of them are proudly my own thoughts or ideas. What they mean by "copyright materials" seems like some of illegal torrent link or video/audio files. I do have some video files uploaded but they are all my own videos.

It seems like it is not only me who had been rejected for the same reason. So I am feeling better after finding that there are more people who are like me.

I don't think there is any way to "prove" who owns a video file. I think they are trying to be risk-averse for unknown zones. None technical people may think my script files or game related posting some kind of "hacking" without knowing what those articles actually mean.

Actually, I think I am still feeling offended. But I guess that's what it is... No pennies for me from Google.

Jun 16, 2013

I was having a trouble connecting Wifi of my IP camera, FI8910W, to my new router, SBG6580.

I got the new router from TimeWarnerCable after I upgraded my Internet speed. The router was actually a good one; I like it a lot. It supports Dual band 5GHz 802.11n and it is a Modem and Wifi router combined device.

However, what I found is that my new router doesn't support old Wifi protocols, 802.11b/g. It only supports 802.11n. Also my IP Camera is not working well as 802.11n, although the tech spec stated "802.11b/g/n".

After wasting a few hours, I ended up using both Wifi routers together: one that covers 5GHz 802.11n and another that covers 802.11b/g/n. Since one of them is using a different frequency, 5GHz, I think they should work without much signal interfere.

Jun 5, 2013

I watch movies through iPad or iPhone. I connect them to TV so that I can enjoy the big screen on my sofa. The WiFi speed is fast enough to carry high definition movies but I found one small problem. Apple is banning AC3/EAC3/DTS movie players. Currently there is no movie play app on iOS that is able to play AC3/EAC3/DTS and access network shared video files.

I think it is some kind of license of patent issues that Apple is complying. The best way for me to work around is to convert those movie files to normal MP3 audio format. So I made a batch file that detect audio format and convert them if necessary.

Audio format can be detected by two softwares: ffmpeg and mplayer. I am using ffmpeg in this article.
There are several transcoding software but I decided to use HandBrake.

Jun 3, 2013

I needed to delete surveillance video files once I have more than certain amount in my hard drive. Let's say I don't want to hold more than 2GByte of video files, which will probably cover one or two month of surveillance video data.

What I need to do with a batch file is like this:

Check how much video files are collected so far.

Compare if the total size is bigger than the size I can afford.

If it is under the space budget, then the batch finishes.

If there are more than I can afford, I want to delete oldest files first.

So I need to figure out what date/time is the oldest.

I will delete files that are at the date/time.

Go to the first step in order to see if I need to delete more.

This was little tricky without installing any external applications.

I had to make two separate batch files due to the syntax limitations on For loop.
One looks like this (c:\batch\delete_surveillance_find_oldest.bat):

Jun 2, 2013

*Edit ( Aug/18/2014 ): I made a C/C++ program that stores the stream video from Foscam IP camera with libvlc. It may work for you better. Check out the article here.

iSpy is a cool open source software that records surveillance video. Although it has several features, the features I was using was recording video/audio when motion is detected. It had been working very well and I was enjoying it a lot.

A little problem was that this software has to be running 24/7 and all of the video data must be streamed from IP camera to the computer that iSpy is running even when there is no motions are detected. Because the motion detection happens on the computer side not on the camera side, the computer must download all the video data from the IP camera via WiFi. I figure the video data isn't too big. It was about 7.3MByte per minute, which is 122KByte/sec.

Today I found that my IP Camera (Foscam FI8910W Wireless IP Camera) has motion detection feature already in it. So it made me thinking that when IP camera detect any motions, it can let my computer know that the recording has to start. This way, the WiFi bandwidth is saved and CPU power that were supposed to be spent on image processing is saved. I don't think neither of them are big deal but I started implementing my idea for fun.

The idea was that my Foscam Camera can upload image files onto FTP server when motion is detected. But it doesn't have any ways to upload video streaming data. So I am recording video streaming data when the FTP connection is requested, while the actual image uploading from the camera will be ignored.
The working scenario is like this:

Foscam Camera monitors the scene.

Foscam Camera detect motions

When any motions are detected Foscam Camera try to connect to FTP server and upload image files.

On the FTP server side, it ignores whatever is sent from Foscam Camera.

But at the time FTP connection is requested, VLC is launched and VLC start recording video streaming data directly from Foscam Camera.

Settings on Foscam Camera

"FTP Service Settings" looks like this.

"FTP Server" is the IP number of the computer that is going store the video streaming data.

"FTP port" is a fake number of this specific task. I will use a number "2121" for this article but you can change it for any numbers.

"FTP User" should be "camera". It is case sensitive like most of FTP login ID.

"FTP Password" doesn't matter for now. But you can make it count by changing the fake ftp server batch file later.

"FTP Mode" should be "PASV" for now.

The way I am explaining here is not going to use a real FTP server. I made a fake FTP server that pretends to be a FTP server. So User/Password doesn't matter at all.

"Alarm Service Settings" looks like this:

"Motion Detection Alarmed" should be checked.

"Upload image on Alarm" should be checked.

"Upload interval (seconds)" can be bigger than 60. This number is something you can tweak later. I don't recommend any number below 60.

These are all settings on Foscam Camera side.
Now when it detect any motions, the Camera will try to upload to the computer. If you have a real FTP server, it will be working. But you will get only images not audio/video.

Now I will explain computer side. This is little hacky and hard to understand but there is no way to harm the computer at all. You will need to install NetCat for windows and VLC for windows. My implementation is all based on Windows but if you are familiar with shell script, you can translate it by yourself because NetCat and VLC should be working on Linux/Mac too.

Once you saved this batch file somewhere, you will need to modify two variables: ServerIP, ServerPort.

ServerIP is the IP number of the computer.

ServerDataPort needs little explanation. When FTP client access to a FTP server, it makes two connections. One is for the command communication and the other is for actual data transporting. They are traditionally port number 21 and 20 respectively. In this article, I am using 2121 and 2120 respectively. If you scroll up, you can see "FTP port" is set to be 2121 on the Foscam Camera and this fake_ftp_server is going to listen to the port number. Then this fake FTP server has to tell what port number is for the data transfer. So the ServerDataPort should be any port number that is not in use. In this article I will use 2120 for the data transfer.

Remember that this FTP server is a fake FTP server. It can check user name and password but for now it cares user name but not password.

The last step of this article is to have a network port listener.
NetCat will keep listening any network access from other machines.
Here is the batch file and save it as "c:\batch\fake_ftp_listener.bat"

@ECHO OFFSET FTP_Port=2121SET FTP_DataPort=2120

REM *** Do not use quotation marks on pathSET FTP_SERVER=C:\batch\fake_ftp_server.batSET NETCAT=C:\batch\nc111nt\nc.exe

You will need to set the Path of batch file and NetCat; FTP_SERVER and NETCAT respectively.
Make sure the Port numbers are matching to other batch files; FTP_Port and FTP_DataPort.

Once you start this batch file, NetCat will start listening two network ports.
Now try to run the batch file.
To test whether it works or not, you can manually connect to the port while fake ftp listener is running.

telnet localhost 2121

If you see a welcome message, type in "USER camera".
If it asks password, then it is working.

As an optional step, you can start it as a hidden process when the windows start.
Save this line as a visual basic script file, "c:\batch\invisible.vbs"

Now you can make a shortcut of the "fake_ftp_listener_hidden_start.bat" in start up folder of your windows.
It is usually C:\Users\[YourUserName]\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup

May 25, 2013

Collada is a XML 3D Model data format. I was looking for a library that can automatically load in structure. I found JAXB that does exactly what I was looking for. However, there was a little trouble generating Java source code.

After struggling for a few hour, I was able to generate JAXB source codes from Collada Schema v1.5.
Here is how I did and it is for anybody who may have the problem I was having.

I got the Collada schema file from here.
At my first try, I got errors:

> xjc.exe -p collada collada_schema_1_5.txt[ERROR] Property "Source" is already defined. Use <jaxb:property> to resolve this conflict.[ERROR] The following location is relevant to the above error

This error is due to name conflict. The name "source" is defined for an attribute and inner element. It can be like this:

<skin_type source="somekindofURL"> <source></source> </skin_type>

The
generated Java source code from JAXB will have a member method
"getSource()" for both the element and the attribute so does the conflict
happens.

This issue can be resolved by providing "binding"
information that simply rename one of them from "source" to something
else. The binding information looks like this:

May 23, 2013

Their webpage didn't give me much information about their service. I was wondering what kind of options I can tweak, how many computers I can use, how can I restore files, how long they will keep my data after my license expires and so on.

What I found after buying it are:

$60/year is only for one computer. I need to add another $60/year for another computer but it seems that they give some discount from 3rd computer.

They delete every backup data as soon as my license expires.

Since my internet uploading speed is not as fast as downloading speed, it seems to take for a while to consume big amount of promised backup storage.

One license covers only one computer to be using the backup software. But it seems that I can download any backup files from any other computers via web user interface. I think this is very useful and otherwise I will be complaining a lot.

The Backup Options:
The backup software runs as a windows tray icon. It doesn't give me complex tweaks. Basically it lets me select which folders to be backed up. It also has some network traffic controlling and CPU utilization checkup to be a less intrusive background task.

The Web Restore:
A picture above is take from their web restore user interface. I can select files to download, then the system generate a ZIP file that includes the files. It seems to take some time to generate the file, which is understandable; when the data size is big, the system will have to spend some time to compress them. The web user interface doesn't seems to allow me to download more than 20GB at once, which is a minor limitation for me.

The Price:
My internet uploading speed is about 0.1MByte/s, which means I can upload 8GByte/day and 3TByte/year. So assuming that I have lots of data to backup 24/7, this is a good deal. Because it is like I am paying $60 for 3TB for the first year and in the next year I will be paying another $60 for another 3TB top of the 3TB I used the previous year.

Comparing to HDD price, I think this service is a good deal although I thought that $60 is little too much at first thought. These days the HDD price is around $90~$140 for 1~3TByte. Although HDD last more than just one year, most of time the limited-warranty is 1/2 years. After 3years later, HDD may stop working and there is a risk to lose all the back up data.

The real question for me at this point is whether I actually have data to backup that much unless I want to securely backup all the HD movie files.

*PS: I tried to access Network Mounted Drive from McAfee Backup software but it didn't show the drive at all. I believe there gotta be a way to trick the software and allow me to access remote files without another license.

May 5, 2013

I have been having a trouble on auto sync feature between my iTunes and my iPhone.
When I was searching Google, many people fixed the same problem by de-selecting "prevent iPods,iPhones, and iPads from syncing automatically" or selecting "Sync with this iPhone over Wi-Fi".

But it didn't fix my problem. One of guys claimed that he fixed the issue by clean uninstalling iTunes with registry cleaner. It didn't work for me too.

Today I fixed the problem. My problem was on Bonjour. It was installed at "C:\Program Files (x86)\Bonjour" and it didn't go away even after I uninstalled every Apple applications. It turned out that it didn't go away because one of DLL files in the folder was still in use by AirDisplay.

Deleting DLL was little tricky because Windows doesn't allow users to alter a file while it is open by any processes. Somehow there were more than AirDisplay was using the Bonjour DLL. I opened CMD window and renamed the DLL in the Bonjour folder and delete the folder. I restarted Windows7 and re-installed iTunes and AirDisplay.

Mileage will vary but this way fixed my problem and I hope this info can help others too.

Yesterday I came up with an interesting idea about programming language.
The idea is from a question how I can separate detail code implementation from abstract logic.

For
example, Java introduced a powerful concept called Interface. An
interface is nothing but a collection of method signature. It cannot
contain any implementation or any variables. First time when I saw it, I thought it is useless because it does nothing for me without any implementation. But the important aspect of interface was to separate the abstract layer of programming code from detail implementation. It also allowed us to reuse the abstract layer a lot more and also allowed us to replace old implementation without affecting other code that are relying on the interface.

I didn't extend my idea from interface but later I found the similarity. And it is easier to understand from the idea of Interface. A function is usually mixed with Logic and implementation details. Let me show us a code example first. This code is taken from a random Google search: http://www.programmingsimplified.com/c-program-copy-file

Although the program itself is short and easy to understand, the issue is that functional logic flow and implementation details are mixed.This problem may not be obvious for short program like this. But when we want the program to be running on multiple platforms, we have to replace implementation details.

First remedy for multi-platform support is usually "#ifdef PLATFORM_A / #else / #endif".
It gets harder to read and gets longer to read.

The point of my view is that even when the implementation details can vary, the logic flow should stay.

When the idea was in my head, it was more interesting than after I wrote them down. The new version of the code is longer and also it utilizes stack memory less than before, which I think not preferable. It will also prevent the compiler from optimizing harder.

But the good part is, as I discussed earlier, the logic flow is reusable so that we can replace the platform dependent code implementation without affecting the logic flow. The "main" function doesn't even need to include stdio.h or stdlib.h.

I may need to find a better example than this to convince anybody else of the usefulness of the idea.