Category Archives: Projects

About

For the Maps Web Application I was tasked with creating a system that could show the location and contact information of all the registered users from the Ida Institute’s website. Moreover, the system had to have a way to show the different groups that users belong to.

People within the countries had to be searchable by name.

There were several important decisions that had to be made to proceed with the development of this project:

What kind of map visualization technology can be used?

How to retrieve user data from the Typo3 CMS that is used for Ida’s website?

How to modify the features in the Maps project

How to show the groups that users belong to?

How to show the members in each of the countries?

How to embed the solution in the Ida’s website and control access to the data.

1. Map Technologies

Looking at the Map display technologies Google Maps usually immediately comes to mind. After examining their API it was decided not to use this solution because the location that was provided by the CMS was way too coarse (only the country was known).

I started looking at the alternatives and came by JVectorMap. This solution is JavaScript based and includes all the maps that the system requires.

2. Data Retrieval

Typo3 CMS provides a functionality to export the list of registered users in a CSV file.

First iteration of the system required manual uploading of the CSV file. This way of providing the updated user data worked but was really cumbersome on the person that had to keep the user’s list in the Maps system up-to-date.

I was not given access to the main database where user’s data is stored and there we no API to retrieve it by other simple means. This is where an idea to move the file retrieval to the Maps application was born.

I managed to login into the Typo3’s admin panel from the Maps code and retrieve the CSV file.

This was done by analyzing how the login process is handled in the CMS’s login page.

After filling in random Username and Password values and trying to login while capturing browser traffic using Fiddler the following WebForm values we determined to be required to perform a login into the CMS’s backend and get access to the people list.

The challenge and the userident fields were changing when trying to login with the same credentials.

Closer inspection of the page source revealed that there is a form that has the required fields set up as hidden inputs.

The challenge value can be extracted by executing a GET request to the login page and parsing the returned text to find the challenge value. The only unclear bit was the way that userident field is used and where it’s value come from. Looking at the form definition there is a call to doChallengeResponse() javascript function before the form is submitted.

It seems that the way the userident field is calculated consists of building a string that contains the username, a MD5 hash value of the password as well as the form’s challenge value separated by the colons (str = document.loginform.username.value+”:”+password+”:”+document.loginform.challenge.value) and calculating a MD5 hash value of the aforementioned string. So I just re-implemented the userident calculation in C#.

After all the required values are ready to be submitted to the Typo3 a POST request is executed using System.Net.Http.HttpClient class and the authentication cookie is used for the next request that retrieves the the CSV file.

The whole process is logged in real time and can be viewed later by the system administrators (emails in the image below are obfuscated for privacy reasons).

3. Settings Panel

To make the web application customizable a simple settings panel was created that allowed to manually upload the CSV file, perform country name mapping, select visible groups (explained later) as well as forcing the data refresh (usually it happens every few days).

4. Groups

The groups are created by parsing the CSV file and extracting all the unique groups that people belong to. All new groups are then stored in the Maps database as separate objects with their visibility set to hidden. This can be changed form the Settings Panel and it makes the group visible in the front page.

After website users click on the group map results are filtered to include only the Ida Members belonging to that particular group.

5. Ida Members

Once a country is selected on the map website users are redirected to a page showing a list of all the members in that country. Since the height of the user boxes could not be predetermined because of different image size and different amount of data the layout had to adapt to different size boxes. Masonry library was used to take care of it. It arranges the grid elements into the most fitting positions.

6. Embedding the Solution

The Maps Web Application was included in Ida CMS’s page as an iframe. The only problem is that the height of the iframe is not know before the page loads. This issue was mitigated using IFrame Resizer libary. After the page is loaded it dynamically adjusts the frame height to match the height of the content inside the iframe.

Conclusion

At the time of writing this post, the project is more or less finished and waiting for the approval to be integrated into the main Ida’s site. The solution for automatically getting the users from the CMS is probably a bit hacky, but without access to data or asking the CMS administrators to build an API for it (which introduces additional cost) this is probably the easiest way to do it.

Log in Screen

To limit access to the back-end system the default forms authentication scheme was used. After logging in the users are redirected to the Newsletters Overview screen.

Newsletters Overview Screen

In the Newsletters Overview screen it is possible to get access to all of the newsletters in the system, edit them as well as create new newsletters.

All of the newsletters are sorted by the year and then by the quarter in the descending order.

Newsletters Edit Screen

The edit screen shows the general information about the newsletter that can be modified (like title, year and quarter) as well as lists all the posts in the newsletter.

The posts can be deleted, edited and there’s a possibility to create new ones. The system currently does not support sorting of the posts after the fact they were added to the system.

Post Screens

Creating and Editing screens for the posts are more or less the same. The provide the functionality to modify the parts of the posts that are visible on the main page. The Content field is using TinyMCE WYSIWYG editor that allows for some customization of the post text.

Other issues

SmarterAsp.net is used to host the solution and they provide a really ugly URL. The company did not want to buy another domain (or use a sub-domain) for the newsletter so I developed a little hack to make it appear that the newsletter is coming form the main Ida’s domain (see the code below).

The PHP script loads the newsletter in an iframe thus masking the URL and making it appear as idainstitute.com/newsletter.

Intro

My girlfriend’s laptop decided to act up… When moving the cable around the charging was being interrupted. First thought – broken cable, but no… It’s worse – broken power jack on the motherboard.

Video

Text transcript

Opening the case

The first part of the dis-assembly is quite simple: remove the cover that protects the RAM, HDD and WiFi module.

Hard disk comes out quite easily as well.

So does the WiFi Module.

To remove the body of the laptop a lot of screws need to be undone. I just followed them from one side and removed them one by one.

Please note, there is a tiny screw in the HDD hole. Without removing it it’s impossible to open up the case. Other tricky place is below the battery. There are two screws there.

Now it’s time to remove the keyboard. One just needs to touch the tabs at the top while lifting the keyboard up gently and it pops out. A wide white ribbon cable needs to be removed before removing the keyboard completely.

And then all the screws can be accessed and removed. One of those screws hold the DVD drive. There are three sneaky screws hiding under the drive.

After using some percussive maintenance and removing a screw or two that I missed, the top cover finally gave in.

Removing the motherboard

To remove the motherboard first gently disconnect the display cable as well as the speakers. There are no tabs holding them, so they just pop out.

Remove the tape, lift up the tab and remove the cable connecting USB hub and the audio jacks. There are 5 screws holding the motherboard, but they are all marked with small white arrows.

Replacing the connector

Now the fun part. Here you can see the broken one as well as the new connector from ebay.

To desolder the old one I am using a desoldering pump that I got from eBay. It costs only 10 bucks but it works like a charm. Just let it heat up and suck away.

A few pins did not want to give up, so I just used some pliers and it gave up. After cleaning up with some solder wick the board is ready for the new connector. Some solder and it looks like new.

Putting it all back together

To put it back together I just followed the dis-assembly in reverse. Motherboard with its 5 screws, then the cables, the lid. After putting on the lid I carefully connected the power button ribbon cable as well as the mouse-pad one. It is important not to forget the three small screws before putting the DVD drive back in. And the one under the HDD. After putting the keyboard back in it’s place the ribbon cable needs to be connected before closing it up.

During this project I had to migrate an already existing flash version of the newsletter to a HTML-based one.

I had the design of the page, but all the features had to be implemented manually.

One of the most interesting features was making the animated half-circles on the page. Since they were supposed to move sideways and the movement had to be interesting, I opted out for making a script (written in JavaScript) that randomizes the lifetime of each of the circles (direction, speed and position).

I have created an HTML version of the Dilemma game that was developed by the Ida Institute. You can check it out at http://idainstitute.com/games/dilemma/. More info about the game here (requires registration).

A really cool thing is that it is self powered and can go to 3.3v keeping it’s current limiting capabilities. I went for the 0-6 Amp version since the price difference is minuscule and it seemed more useful in the future. According to the documentation it can handle 20 W or more with the larger heat-sink. Who does not like having some extra leeway…

Please see the full review as well as build process in my YouTube video:

This post is about a project that took ages to built and I manage to loose in an unfortunate circumstances…

I always wanted to build a clock. And this is what came up…

The Idea

Create a modular clock that has interchangeable displays.

The Execution

Main controller

The main module for the clock is an ATmega328 microcontroller with a DS1307 Real Time Clock. The LCD displays current date and temperature as well as shows the setup menu.

Interaction with the clock is done by one rotary encoder that has a built-in push button. The state diagram for the menu can be found below.

Display Controllers

Time display (described below) is controller by a series of 74HC595 shift registers. Since there is a need for two digits per segment (by segment I mean hours, minutes and seconds) there are 6 of them in total connected in series. To update the time display I have to shift out the current state of the hours, minutes and seconds every second. Since the update rate is relatively slow no stress is applied to the microcontroller and it can do other tasks like displaying temperature readings, etc.

Time display

Time is displayed on a self-built 7 segment displays. There are 3 LED’s for the horizontal axis and 4 for the vertical one. By testing various combinations I found that this setup was the most appealing to my eyes. LED’s in a segment are connected in parallel and routed to the input connector at the bottom of the board (color-coded cable in the photo). The only problem I had with the displays was that they were really bright even on the lowest brightness setting (that can adjusted through the menu). To mitigate this problem I added some plexiglass that I painted white to diffuse the light a bit. Note to myself for the future projects: do not use clear LED’s for such applications…

More photos

A short clip showing the testing of the time display

Conclusion

I had a lot of fun working on this project. It was a really nice learning experience and I am planning on rebuilding the project in the future with some improvements.

I’ve been looking for a while now to find a way to switch between multiple audio sources and output them on my speaker system.

Finally, I decided that the simple way is the best way :), so there’s what I came up with.

This is a 3P4T switch with four 3.5mm stereo female audio jacks acting as input and a 3.5mm stereo female audio jack that outputs the selected channel to the speakers. The knob points to the channel that is selected. Audio sources in this photo is my main PC and a Media laptop used for watching videos. There’s still two unused inputs left for connecting my phone/tablet/etc.

Now some photos from the build process:

I’ve spend approx. 100 DKK (17.5USD) for the parts (box, switch, knob, audio cable) that I did not have in my spare parts bin, but also had lots of fun building it, so, to my mind, it was worth the price.

Intro

I got a ST7735 based lcd from ebay and decided to play with it a bit.

Setup

Arduino UNO

1.8″ TFT LCD from ebay

Some jumper cables

Code

All the guidelines for connecting the LCD to the Arduino can be found here: http://www.ladyada.net/products/18tftbreakout/, but there is one thing to note. In the tutorial there is a pin called “dc” (Data/Command) (thanks to sza2) (I am not sure what that stands for) , but on my LCD it’s named “RS” (register select). So, to make it work, just ignore the difference in naming.

Video

Problem

Let’s start with a problem…

In order to have my phone as well as my BlueTooth audio headset working I had to carry two different chargers to workplace. Moreover, I needed to find an unused socket to plug them in. And that’s really annoying…

Idea

I could see two possible solutions:

Buy a charger that uses batteries and provides 5V to charge my devices

Build one of my own

And, of course, I went with choice #2. I had some Lithium Ion batteries lying around. And this was the starting point of LIBMS.

LIBMS

It would be crazy to just power devices through some voltage regulator. LiIon cells do not like to be discharged below 3V and without some additional circuitry it was impossible to check whether they were OK to continue providing power to the devices. (Well, I could carry a multimeter and measure each of them from time to time, but that would be crazy :))

I found some IC’s that are capable of monitoring LiIon batteries, but where’s the fun in that…

I had some Atmega328 chips laying around. They have a 6 channel 10-bit ADC (there is actually only one ADC, but built-in multiplexer expands the inputs to 6). Awesome.

Batteries

I connected 3 of the batteries in series and then that block with another block of three batteries in parallel. V1..V6 show my tap points that are used to measure each cells voltage. E.g.

V(G2) = V2 – V1

V(G3) = V6 – V(G2)

…

V(G6) = V6 – V(G5)

Voltage regulators

At the beginning of this project I had some problems with voltage regulation. I powered the microcontroller (uC) directly from the main voltage regulator (LM338), but when a device was connected that requires a lot of current the voltage suddenly dropped and uC restarted. Not good…

I solved this problem by introducing secondary LM7805 voltage regulator that is used to power the uC and LM338 takes care of the USB ports. Using this structure provided me with a cool way for controlling the USB ports from uC.

Browsing through the LM338’s data-sheet I found an example where the regulator can be controller using a TTL signal. Perfect. Even though it does not shut down completely (there is still ~1.25V on the output), it does the job.

Sensing part

Because Atmega328 can measure up to 5V max, I had to scale down the voltages of the cells. To do that I used voltage dividers. There are 5 of them, connected to ports A0..A4. Resistors (with values specified) are R1..R10 in the schematic.

Schematic & PCB layout

*Click on the images for hi-res version.

There is a 10K pull-up resistor (not shown in schematic) connected from BC337 base to Vcc to ensure that Voltage regulator stays off while uC is starting up.