Category Archives: Other

As can be seen by the post dates, I’ve experienced another one of my blogging hiatuses. This was due mostly to going into crunch mode trying to finish up emissary RT – I really needed to get it wrapped up before September, as I needed to have my schedule wide open by the start of the month. The reason why – I got accepted to grad school! I’ll be starting the program to get my Masters in Computer Science in a few days, and needed to get this final item checked off my list. I’m very excited for school, but equally happy to be finished with emissary RT – it was a fun project that I’ve long had the idea for, but after 2 years of development, I was ready for it to be complete.

In other news, I’ve really been diving back into my gaming roots lately. I recently finished listening to Masters of Doom on audiobook (biography of John Carmack and Romero – get it NOW if you haven’t read it already!), and along with bringing back a HUGE slew of memories from gaming in the 90s (shareware like Commander Keen, BBSes, the start of the Internet, etc), it was also incredibly inspiring to hear the story of some passionate developers following their dreams and love of development. Along with this, I also finally started playing with Unity, which I’ve been meaning to try for a while. Long story short, I am completely hooked on the game engine, and incredibly ramped up to start a new game (it’s been too long since my last one), so along with working more on SynthNet, this will be my next big project. More details soon!

After many, many years (I’ve lost count at this point) of faithful service , I’ve finally refreshed the Synthetic Dreams website into something a little more modern and functional. Take a look if you’ve got a moment, it’s built on Drupal (of course), and features a responsive design for those browsing on the go.

Additionally, after being in development for almost 2 years, I’ve finally finished emissary RT – an ODBC driver that allows you to access a whole slew of things, from your file system to DHCP and DNS. The upshot of this is allowing you to use SQL (or the GUI in ODBC apps) to manipulate files and services in very powerful and automation-friendly ways. You can check out the full details on the Synthetic Dreams site as well.

I realized while responding to some comments that I completely forgot to mention some exciting news! Last month, I was fortunate enough to have an article featured on the Qualcomm Spark website, “Can We Grow Artificial Intelligence?” It explores some of the capabilities we currently have of emulating DNA and biological growth, and incorporating these abilities into our normal programming tools to develop all sorts of AI. I had a lot of fun writing it, as well as reading the other articles featured on the site. So many exciting technologies on the horizon (or already here!)

Though I have fun working on SynthNet and other projects at night, during the day I fill the role of mild-mannered network administrator at the Manchester-Boston Regional Airport (actually, the day job is quite a bit of fun as well). One of the ongoing projects I’ve taken on is adding all of our various Intranet-oriented services into a single platform for central management, easier use, and cost effectiveness. As mentioned in a previous article (linked to below, see NMS Integration), I knew Drupal was the right candidate for the job, simply due to the sheer number of modules available for a wide array of functionality, paired with constant patching and updates from the open source community. We needed a versatile, sustainable solution that was completely customizable but wasn’t going to break the bank.

The Mission

The goal of our Drupal Intranet site was to provide the following functionality:

PDF Document Management System

Categorization, customized security, OCR

Desktop integrated uploads

Integration with asset management system

Asset Management System

Inventory database

Barcode tracking

Integration with our NMS (Zenoss)

Integration with Document Management System (connect item with procurement documents such as invoices and purchase orders)

Amazingly enough, the core abilities of Drupal covered the vast majority of the required functionality out of the box. By making use of custom content types with CCK fields, Taxonomy, Views, and Panels, the typical database functionality (entry, summary table listings, sorting, searching, filtering, etc) of the above items was reproduced easily. However, specialized modules and custom coding was necessary for the following parts:

Customized Security – Security was achieved for the most part via Taxonomy Access Control and Content Access. TAC allowed us to control access to content based on user roles and categorization of said content (e.g. a user who was a member of the “executive staff” role would have access to documents with a specific taxonomy field set to “sensitive information”, whereas other users would not). Additionally, Content Access allows you to further refine access down to the specific node level, so each document can have individual security assigned to it.

OCR – This was the one of the few areas we chose to delve into a commercial product. While there are some open source solutions out there, some of the commercial engines are still considerably more accurate, including the one we choose, ABBYY. They make a Linux version of the software that can be driven via the shell. With a little custom coding, we have the ABBYY software running on each PDF upload, turning it into an indexed PDF. A preview of the document is shown in flash format by first creating a swf version (using pdf2swf), then using FlexPaper/SWF Tools.

Linking Documents – This was performed with node references and the Node Reference Explorer module, allowing a user friendly popup dialogs to choose the content to link to.

Desktop Integration – Instead of going through the full steps of creating a new node each time, choosing a file to upload, filling in fields, etc, we wanted the user to be able to right click a PDF file on their desktop, and select “Send To -> Document Archive” from Windows. For this, we did end up doing a custom .NET application that established an HTTP connection to the Drupal site and POSTed the files to it. Design of this application is an article in itself (maybe soon!).

Barcoding – This was the last place we used a commercial product simply due to the close integration with our barcode printers (Zebra) – we wanted to stick with the ZebraDesigner product. However, one of the options in the product is to accept the ID of the barcode from an outside source (text/xml/etc), so this was simply a matter of having Drupal put the appropriate ID of the current hardware item into a file and automating ZebraDesigner to open and print it.

NMS (Zenoss) Integration – The article of how we accomplished this can be found here.

Automated Switch Configuration Backups and Network Tracking – This just took a little custom coding and was not as difficult as it might seem. Once all our network switches were entered into the asset management system and we had each IP address, during the Drupal cron hook, we had the module cURL the config via the web interface of the switch by feeding it a SHOW STARTUP-CONFIG command (e.g. http://IP/level/15/exec/-/show/startup-config/CR) – which was saved and attached to the node. Additionally, we grabbed the MAC database off each switch (SHOW MAC-ADDRESS-TABLE), and parsed that, comparing the MAC addresses on each asset to each switch port, and recording the switch/port location into each asset. We could now see where each device on the network was connected. A more detailed description of the exact process used for this may also be a future article.

Help Desk – While this could have been accomplished with a custom content type and views, we chose to make use of the Support Ticketing Module, as it had some added benefits (graphs, email integration, etc)

Automated Gathering of Hardware Info – For this, we made use of a free product called WinAudit loaded into the AD login scripts. WinAudit will take a full accounting of pretty much everything on a computer (hardware, software, licenses, etc) and dump them to a csv/xml file. We have all our AD machines taking audit during logins, then dumping these files to a central location for Drupal to update the asset database during the cronjob.

Active Directory Integration – The first step was to ensure the apache server itself was a domain member, which we accomplished through the standard samba/winbind configurations. We then setup the PAM Authentication module which allowed the Drupal login to make use of the PHP PAM package, which ultimately allows it to use standard Linux PAM authentication – which once integrated into AD, includes all AD accounts/groups. A little custom coding was also done to ensure matching Drupal roles were created for each AD group a user was a part of – allowing us to control access with Drupal (see #1 above) via AD groups.

There was a liberal dose of code within a custom module to glue some of the pieces together in a clean fashion, but overall the system works really smoothly, even with heavy use. And the best part is, it consists of mainly free software, which is awesome considering how much we would have paid had we gone completely commercial for everything.

Please feel free to shoot me any specific questions about functionality if you have them – there were a number of details I didn’t want to bog the article down with, but I’d be happy to share my experiences.

In this uber-connected, social media driven world, it seems like the time between when an idea is born and when it completely saturates the Internet twenty times over is almost nil. While this does mean seeing Dramatic Chipmunk and Nyan Cat until the point of retinal damage, it also has the benefit of introducing the masses to really cool ideas and projects from all around the globe. It means more people sharing their creations, which is a win-win for everyone.

Because of this mass spread of information, it always surprises me how many people are unfamiliar with the demoscene. Having grown up a Commodore 64 (and later Amiga) kid who hung out on BBSes, intros and demos were always a part of my computer world. At that time, they were amazing, mysterious creations, made by programmers with futuristic-sounding handles from far away countries. As I grew older, I started to not only befriend many sceners, but also think more about both the Computer Science and art that actually went into these – and my amazement only increased. Now I do everything I can to show off these programmatic, musical, and artistic feats to anyone who will watch.

The Scene

To quote Wikipedia, “The demoscene is a computer art subculture that specializes in producing demos, which are audio-visual presentations that run in real-time on a computer. The main goal of a demo is to show off programming, artistic, and musical skills.” Originally, they started as shout-outs and other introductions in game cracks on 8-bit computers, showing off programming skill. They quickly bloomed into an entire culture of demogroups, competitions, parties, boards, etc – and is still going strong today with a strong European core (though still prevalent in the US!). I encourage you to learn more about all the awesome history behind the scene – there is more than can be covered in one blog post.

Favorite Demos

While the history is interesting, what is more important are the demos themselves! Below I’ve included 4 of my favorite demos. The first two are 64K PC demos. When I say 64K, I mean the entire demo is 64K big. Graphics, music, code – everything. This is procedural programming on steroids – artistic and algorithmic wonderment.

The second two are for the Commodore 64. While they are more limited by the hardware, the talent still shines through. The first is a great example of an amazing musical score, and the second is unbelievable coding and use of the C64 hardware, making it look more like a 16-bit machine.

This is just a taste of what has come out over the years – I encourage you to take a look at sites like pouet.net and The Commodore Scene Database for some more examples. Be prepared to be amazed!

Recently, I started talking with my girlfriend about the idea of writing a life plan. The idea is similar in nature to a business plan, but instead of outlining the structure, mission statements, and strategies of a financial venture, you’re focused on the values, goals, and eventualities of your life as a whole. I’ve researched a bit online, and the more I thought about it, the more I realized what a completely awesome tool a life plan could be – not only for organizing your life, but just the process of writing one can really illuminate and flesh out life-goals. More importantly though, as I realized by talking with my friends, it can truly be a living document, one that grows over time as life, values, and situations change.

Though I am only in the planning stages now of what I want to include in my plan, I know before I put a single word down that there are two items that I will inevitably focus on. The first is one of my true passions in life – creating. Specifically, creating through computer science – games, AI, network utilities, or anything. But ultimately I know this isn’t truly fulfilling. I read article after tweet after blog post about software development and computer science – and some writing inspires me, and some falls flat. It took me a while to figure out why, and as of late I realize more why that is. Which brings me to the second item I will focus on – helping the world. If I have a limited time on this big, blue globe, I want to do whatever I can to ensure that hopefully, at least in some small part, my creations will make the life a better place. This – and making connections with other people who want to use their awesome skills to do some serious good! I’m lucky enough to lots of friends with this attitude, and I’d love to make more.

Resources

To say there are a lot of amazing organizations out there changing the world on a daily basis would be an understatement – our lives change constantly with the evolution of social networks, mobile devices, and interconnectivity. And while many of these changes attack very real problems and improve quality of life, there is still infinite amounts of space to effect positive change – still countless opportunities to do good. And I think it’s important to deliberately focus on these items as a core goal. I’ve recently begun to search online for resources and other like-minded buddies to help in this quest – and I’ve found a number in academia, as well awesome sites like TED that have some truly brilliant people focused on these very issues.

If you know of any other resources that talk about helping the world through computer science or other technology-driven philanthropy, please feel free to send them this way!

Or if you have any experience with writing a life plan or steps you’ve taken to clarify goals for yourself, please feel free to drop me a line!

I know there are other people much smarter than me who have tackled these areas before, so I’d love any guidance or tips. I hope to continue to post on these subjects as I learn more and make further connections.

James Pearn at artificialbrains.com was nice enough to include SynthNet in his list of resources related to artificial intelligence. Check out his site if you get a moment, it serves as a well laid-out directory of many neural network and other artificial intelligence projects going on around the world, as well as job listings. Very cool site – thanks James!

One of the more daunting tasks of managing a larger network is keeping track of all your devices – both physically, and from a network monitoring perspective. When I arrived on the job 3 years ago, the first major task I laid down for myself was implementing both an asset management system, as well as a network monitoring system, to ensure we always knew what we had, and if it was functioning properly.

I decided almost immediately that Drupal was the right candidate for the job of asset management. There are a number of commercial IT/helpdesk systems out there which work great, but they are usually fairly expensive with recurring licensing costs, and my history with them has always been shaky. Plus, I find myself not always using all the functionality I paid for. I knew with my Drupal experience, I could get something comparable up in almost no time – this is not a discredit to IT packages, but moreso the power of the Drupal framework.

Network Monitoring – Cue Zenoss

Now that I had the hardware DB taken care of, I needed a NMS for monitoring. Originally I was planning on Nagios, but a contractor who works for us (now friend) had introduced me to Zenoss, another open source alternative. Zenoss is awesome – is absolutely has its quirks, and is not the most intuitive system to learn, but once things are up and running it’s great – and tremendously powerful. So the choice was made.

Now – I had both pieces, but I absolutely hate entering data twice, and the interoperability guy in me loves integrating systems. So I decided to write a script that would sync our Drupal database with Zenoss. Drupal would serve as our master system, and any hardware we entered into it would automatically port over to Zenoss. Any changes or deletions we made (IP address, location, name, etc) would sync over as well.

The below script performs this synchronization. Some warnings up front – I’m not a Python guy by any means, I specifically learned it for this script, so I apologize for any slopping coding or obvious Python-y mistakes. I’ve tried to thoroughly comment it to document how to use it and how it works. Hopefully it can help some others out as well!

# Description: Sync devices to be monitored from Drupal to Zenoss
#
# Setup Work: Create a (or use an existing) content type that houses your hardware items to be monitored.
# They should have CCK fields for the IP address of the device, the name, and the type of
# device it is. The device type will determine the Zenoss class the script adds it to, and hence
# the kind of monitoring it will receive (e.g. Linux server, switch, ping only, etc)
#
# Additionally, in Zenoss, create a custom property field that will house the nid of the Drupal
# node. This serves as the foreign key and will be used to link the item in Drupal to its entry in Zenoss
#
# Usage: This script should be run from zendmd, and may be run once or periodically. We run it every 20 minutes from
# a cron job.
# It will create new entries in Zenoss for items not yet imported, delete ones that no longer exist in
# Drupal (it will only delete ones that were originally imported from Drupal), and will update ones that have
# been updated (type, IP, location, etc).
#
# Note: Excuse all the extra commits - we experienced some issues with data not being saved, and I threw some extra in
# there - they're almost definitely not necessaryimport MySQLdb
# Take a taxonomy term from Drupal identifying the type of monitoring to be done,
# and convert it to the appropriate Zenoss class path. Update these to whatever terms
# and Zenoss class paths that make sense for your environment. We setup ones for
# Linux and Windows servers, switches, waps, UPSes, PDUes, etc, as can be seen.
def getClassPath(passType):
if passType.lower() == "windows":
return "/Server/Windows"
elif passType.lower() == "linux":
return "/Server/Linux"
elif passType.lower() == "switch":
return "/Network/Switch"
elif passType.lower() == "mwap":
return "/Network/WAP/Managed"
elif passType.lower() == "uwap":
return "/Network/WAP/Unmanaged"
elif passType.lower() == "ups":
return "/Power/UPS"
elif passType.lower() == "pdu":
return "/Power/PDU"
elif passType.lower() == "camera":
return "/Camera"
elif passType.lower() == "cphone":
return "/Network/Telephone/Crash"
elif passType.lower() == "sphone":
return "/Network/Telephone/Standard"
elif passType.lower() == "printer":
return "/Printer"
elif passType.lower() == "converter":
return "/Network/Converter"
elif passType.lower() == "ping":
return "/Ping"
return "/Ping"
# Connect to Drupal's MySQL DB (Replace these values with the appropriate ones for your system)
imsConn = MySQLdb.connect(DRUPAL_MYSQL_SERVER, MYSQL_USER, MYSQL_PASSWORD, MYSQL_DB)
imsCursor = imsConn.cursor()
# Execute the query to grab all your items to be monitored. In our case, we have a node type called "hardware" that had CCK fields identifying the IP address,
# the type of hardware (a taxonomy term that dictated the Zenoss class of the item - see getClassPath above), a physical location, etc.
# You'll want to change the specific table/field names, but the inner join will probably stay, as you'll want to grab both the node and CCK fields that belong to it.
imsCursor.execute("""
SELECT node.nid, content_type_hardware.field_hardware_dns_value, content_type_hardware.field_hardware_location_value, content_type_hardware.field_hardware_ip_value, content_type_hardware.field_hardware_monitor_type_value, content_type_hardware.field_hardware_switchname_value, content_type_hardware.field_hardware_switchport_value
FROM node
INNER JOIN content_type_hardware ON node.nid = content_type_hardware.nid
""")
# Loop through all returned records - Check for additions, changes, and removals
while (1):
#tempRow is our current hardware item record
tempRow = imsCursor.fetchone()
if tempRow == None:
# No more entries, break out of loop
break
else:
# Search Zenoss records for the nid of the hardware item. A custom field will need to be created in Zenoss to serve
# as this foreign key. In our case, we used MHTIMSID - but you can use anything you'd like - just be sure to create the field in Zenoss.
found = False
for d in dmd.Devices.getSubDevices():
if d.cMHTIMSID != "":
if int(d.cMHTIMSID) == tempRow[0]:
found = True
break
if found == False:
# Hardware item not found, add it if it's monitored
if tempRow[4] != None:
dmd.DeviceLoader.loadDevice(("%s.yourdomain.com" % tempRow[1]).lower(), getClassPath(tempRow[4]),
"", "", # tag="", serialNumber="",
"", "", "", # zSnmpCommunity="", zSnmpPort=161, zSnmpVer=None,
"", 1000, "%s (%s - %s)" % (tempRow[2], tempRow[5], tempRow[6]), # rackSlot=0, productionState=1000, comments="",
"", "", # hwManufacturer="", hwProductName="" (neither or both),
"", "", # osManufacturer="", osProductName="" (neither or both).
"", "", "", #locationPath="",groupPaths=[],systemPaths=[],
"localhost", # performanceMonitor="localhost',
"none")
tempDevice = find(("%s.yourdomain.com" % tempRow[1]).lower())
tempDevice.setManageIp(tempRow[3])
commit()
# Save nid to Zenoss record (to serve as foreign key) for syncing
tempDevice._setProperty("cMHTIMSID","MHTIMS ID","string")
tempDevice.cMHTIMSID = tempRow[0];
commit()
else:
# Hardware item found - delete, update, or do nothing
if tempRow[4] == None:
# Delete if not set to monitor
dmd.Devices.removeDevices(d.id)
else:
# Update DNS and IP to current values
if d.getDeviceName() != ("%s.yourdomain.com" % tempRow[1]).lower():
d.renameDevice(("%s.yourdomain.com" % tempRow[1]).lower())
if d.getManageIp() != tempRow[3]:
d.setManageIp(tempRow[3])
commit()
# Change class if not set to "Manual" (We setup a taxonomy term called "Manual" that would turn off automatic Zenoss class selection during syncing
# and allow us to manually specificy the class of the device.
if tempRow[4] != "Manual":
d.changeDeviceClass(getClassPath(tempRow[4]))
commit()
# Update comments (location change)
d.comments = "%s (%s - %s)" % (tempRow[2], tempRow[5], tempRow[6])
commit()
# Save any missed changes
commit()
# Close connection to database
imsCursor.close()
imsConn.close()

After finishing up phase 1 of SynthNet, I came to the conclusion that I really missed updating the blog. When I get involved in a project, I tend to get wrapped up in it (more accurately – completely and ridiculously obsessed where it takes over my life) and other things drop off the radar. However, I’ve decided I want to make a real effort to not get AS wrapped up in projects, and remember to give the blog some love.

New and Improved!

As I went to write my first article after recording the SynthNet video, I also noticed the blog was looking a little tired. They’d also made a number of improvements in WordPress since when I first installed everything, so I decided to take the leap, get a shiny new template, and put some new life into it. I think it’s definitely an improvement – hope you enjoy it!

One of the best things about being involved in the world of technology, besides getting a front row seat to all the amazing advancements made every day, is meeting and talking with the creative people who make the magic happen. I think I’m especially lucky, having strong ties to a range of different areas such as networking and development, to have met a diverse mix of very talented people.

Web Developer Extraordinaire

To say business exists in a social media world where online presence and reputation is important would be the understatement of the century. Companies today live and die by their ability to harness the power of the web. And while there are many developers out there, a true burden lies in finding talented and experienced ones. Not only does Leah fall into this camp, combining expert design skill with seasoned web development knowledge, but she possesses something that many in the industry don’t – a real love and respect for what her customers are trying to accomplish with their website. This truly shines through both in her work, and how she treats her clients. It translates into a special website that speaks its goals and connects to its visitors like no other site could. It is the difference between a good looking site and a truly powerful site.

The Proof is in the Pudding

I’ve known Leah for a number of years, having had the privilege of working with her on a number of projects professionally – and her sites continue to really impress me. Some excellent examples of recent projects: Be Irreplaceable | Donna Heart.

I love these examples, as they show how she has taken a general framework like WordPress, and turned it into a beautiful site that really communicates the site’s message. They feel personable and comfortable when you visit them, unlike a lot of cold and bland sites out there. They have that truly personal touch which is key to connecting with the audience.

So if you’re looking to build a new website for your business, or need to re-imagine the one you already have, I really suggest keeping Leah Creates in mind. She is amazing at both what she does, and how she does it – something setting her apart from so many other development houses out there.