Tech Roots » C#http://blogs.ancestry.com/techroots Ancestry.com Tech Roots BlogsTue, 31 Mar 2015 14:55:00 +0000en-UShourly1http://wordpress.org/?v=3.5.2Monitoring progress of SOA HPC jobs programmaticallyhttp://blogs.ancestry.com/techroots/monitoring-progress-of-soa-hpc-jobs-programmatically/ http://blogs.ancestry.com/techroots/monitoring-progress-of-soa-hpc-jobs-programmatically/#commentsFri, 17 Oct 2014 14:15:27 +0000Chad Gronemanhttp://blogs.ancestry.com/techroots/?p=2873Here at Ancestry.com, we currently use Microsoft’s High Performance Computing (HPC) cluster to do a variety of things. My team has multiple things we use an HPC cluster for. Interestingly enough, we don’t communicate with HPC exactly the same for any distinct job type. We’re using the Service Oriented Architecture (SOA) model for two of… Read more

]]>Here at Ancestry.com, we currently use Microsoft’s High Performance Computing (HPC) cluster to do a variety of things. My team has multiple things we use an HPC cluster for. Interestingly enough, we don’t communicate with HPC exactly the same for any distinct job type. We’re using the Service Oriented Architecture (SOA) model for two of our use cases, but even those communicate differently.

Recently, I was working on a problem where I wanted our program to know exactly how many tasks in a job had completed (not just the percentage of progress), similar to what can be seen in HPC Job manager. The code for these HPC jobs uses the BrokerClient to send tasks. With the BrokerClient, you can “fire and forget”, which is what this solution does. I should note that the BrokerClient can retrieve results, after the job is finished, but that wasn’t my use case. I thought there should be a simple way to ask HPC how many tasks had completed. It turns out that this is not as easy as you might expect, when using the SOA model. I couldn’t find any documentation on how to do it. I found a solution that worked for me, and I thought I’d share it.

HPC Session Request Breakdown, as shown in HPC Job Manager

With a BrokerClient, your link back to the HPC job comes from the Session object used to create the BrokerClient. From a Scheduler, you can get your ISchedulerJob that corresponds with the Session by matching the ISchedulerJob.Id to the Session.Id. My first thought was to use ISchedulerJob.GetTaskList() to retrieve the tasks and look at the task details. It turns out that for SOA jobs, tasks do not correspond to requests. The tasks don’t have any methods on them to indicate how many requests they’ve fulfilled, either.

My solution was found while looking at the results of the ISchedulerJob.GetCustomProperties() method. I was surprised to find the solution there, since the MSDN documentation states that this is “application-defined properties”.

I found four name-value pairs which may be useful for knowing the state of tasks in a SOA job, with the following keys:

“HPC_Calculating”

“HPC_Caclulated”

“HPC_Faulted”

“HPC_PurgedProcessed”

I should note that some of these properties don’t exist when the job is brand new, with no requests sent to it yet. Also, I was disappointed to find no key corresponding to the “incoming” requests, since some applications might not be able to calculate that themselves.

With that information, I was able to write code to monitor the SOA jobs.

With all that said, I should also say that our other SOA HPC use case monitors the state of the tasks, and is capable of more detailed real-time information. We do this by creating our own ChannelFactory and channels. By using that, the requests are not “fire and forget” – we get results back from each request individually as it completes. We know how many outstanding requests there are, and how many have completed. If we wanted to, we could use the same solution presented for the BrokerClient to find out how many are in the “calculating” state.

One last disclaimer: These “Custom Properties” are not documented, but they are publicly exposed. Microsoft could change them. If they ever do, I hope they would consider it a breaking change, and document it. There are no guarantees of that, so use discretion when considering this solution.

]]>http://blogs.ancestry.com/techroots/monitoring-progress-of-soa-hpc-jobs-programmatically/feed/2Maintaining Balance by Using Feedback Loops in Softwarehttp://blogs.ancestry.com/techroots/maintaining-balance-by-using-feedback-loops-in-software/ http://blogs.ancestry.com/techroots/maintaining-balance-by-using-feedback-loops-in-software/#commentsTue, 29 Jul 2014 17:54:00 +0000Chad Gronemanhttp://blogs.ancestry.com/techroots/?p=2558Maintaining Balance by Using Feedback Loops in Software Feedback is an important part of daily life. Used wisely, feedback loops can help us make better decisions, resulting in overall improvements. Feedback loops are useful in all sorts of situations, from relationships to what we eat for dinner. Software can also be made to take advantage… Read more

Feedback is an important part of daily life. Used wisely, feedback loops can help us make better decisions, resulting in overall improvements. Feedback loops are useful in all sorts of situations, from relationships to what we eat for dinner. Software can also be made to take advantage of feedback loops.

Feedback, Feedback loops, and Apple Pie

The difference between feedback and a feedback loop is an important one. To illustrate, let’s say I get a new apple pie recipe I want to try out. I follow the directions exactly, and the pie comes out burnt. I see that it’s burnt (feedback), so I throw it out rather than eat it. But I really like apple pie, so I the next day I try again, with the same result. Using feedback alone, I can throw the pie out again and again until the day I die. What is needed is a feedback loop. A feedback loop is where existing result(s) are used as input. Where feedback prevented me from eating the burnt pie, a feedback loop can be used to make adjustments to the baking time and/or temperature. After a few iterations of learning from the feedback, I’ll be able to enjoy a tasty apple pie, cooked to perfection.

Feedback in Software

There are many forms of feedback in software, including exceptions, return values, method timings, and other measurements. Whether it will be useful to create a feedback loop in your application is up to you – often times, there is little value in creating an automated feedback loop. However, there are situations where feedback loops can be very helpful. Software that is resource-bound or dependent on external systems are examples of software that can benefit from feedback loops. Here at Ancestry.com, much work has been done to create feedback loops based on service level agreements between dependent systems. We’ve seen a lot of benefit, but we’ve also had a learning experience or two. If you’ve ever heard screeching because of feedback with a microphone, you know that not all feedback loops produce improved results. Yes, it’s possible to produce bad feedback loops in software. This can happen with a single feedback loop, but it’s especially true if you have multiple feedback loops — multiple feedback loops with bidirectional interactions can quickly become a debugging nightmare. The more you can compartmentalize the moving parts and limit interactions, the easier maintenance will be.

A real-world example

One of the projects I work on is a system that finds and fixes problems. Most of the jobs this system runs are low priority. The system uses a shared compute cluster to do most of the work, and we don’t want it to use the cluster if it is loaded up with higher priority work. Rather than queue up jobs in the cluster for days at a time, a feedback loop is used to determine whether jobs should run on the cluster or local processors. The system monitors the cluster by sending a “ping” type task periodically, and tracking that feedback (timings, exceptions, etc.) along with the feedback from real tasks. With that feedback, the system determines when the cluster should be used, and will dynamically change the compute destination based on that feedback. Meanwhile, another feedback loop that tracks how much work is queued up in memory, and automatically adjusts how quickly work is produced. We have other feedback loops around the data sources, to take action if we suspect this application is slowing down production systems. The entire system works pretty well.

My team has seen improved performance and stability as feedback loops have been judiciously added. We’ve found them to be worthwhile on any system we’ve applied them to, critical or not. While it seems there’s always more we can do, it’s nice to have systems do some self-tuning to keep things efficiently running.

]]>http://blogs.ancestry.com/techroots/maintaining-balance-by-using-feedback-loops-in-software/feed/0Utah Code Camp 2014 – A Success for Ancestry.com Tech Team and Whole Communityhttp://blogs.ancestry.com/techroots/utah-code-camp-2014-a-success-for-ancestry-and-the-whole-community/ http://blogs.ancestry.com/techroots/utah-code-camp-2014-a-success-for-ancestry-and-the-whole-community/#commentsThu, 20 Mar 2014 18:27:12 +0000Mitchell Harrishttp://blogs.ancestry.com/techroots/?p=2099Utah Code Camp 2014 came and went this weekend. More than 850 people attended and with more than 70 sessions, it was the largest code camp in Utah history. Thanks to Pat, Craig, Nate, and Kerry of Utah Geek Events for putting it all on. Ancestry.com participated in a pretty big way. In addition to the… Read more

]]>Utah Code Camp 2014 came and went this weekend. More than 850 people attended and with more than 70 sessions, it was the largest code camp in Utah history. Thanks to Pat, Craig, Nate, and Kerry of Utah Geek Events for putting it all on.

Ancestry.com participated in a pretty big way. In addition to the many Ancestry.com employees in attendance, we had three speak. Bressain Dinkelman presented a session titled, “Yes, You Belong Here,” about the Imposter Syndrome in the tech world, and how to overcome it. Craig Peterson presented a session titled, “High Performance Web Services with Apache Thrift.” I presented a session called, “Making Your Own Domain Specific Language,” and also was called in last minute pinch to hit speak on RavenDB for a presenter that got pneumonia.

In addition to employees attending and speaking at the event, Ancestry.com also sponsored the event. Did you find our spot next to the elevators on the first floor? We were handing out T-Shirts, hand sanitizer, pens, candy, and evangelizing how great it is to work here. We’re proud to be a sponsors to of Utah Code Camp, and hope for another great camp next year.

]]>http://blogs.ancestry.com/techroots/utah-code-camp-2014-a-success-for-ancestry-and-the-whole-community/feed/0Announcing Daisy, An Ultra Lightweight, Open Source Rules Enginehttp://blogs.ancestry.com/techroots/announcing-daisy/ http://blogs.ancestry.com/techroots/announcing-daisy/#commentsThu, 23 Jan 2014 22:24:57 +0000Mitchell Harrishttp://blogs.ancestry.com/techroots/?p=1352We just released a new open source package here at Ancestry: Daisy. Daisy is a business writeable domain specific language for business rules. It lets business experts write the rules, and software developers define how they are implemented. Daisy itself is domain agnostic, but allows domain experts and software developers to create the rule domain together. An example… Read more

]]>We just released a new open source package here at Ancestry: Daisy. Daisy is a business writeable domain specific language for business rules. It lets business experts write the rules, and software developers define how they are implemented. Daisy itself is domain agnostic, but allows domain experts and software developers to create the rule domain together. An example rule is:

Any User
Is Active
Has Account
Balance is less than 0

OR Balance is greater than 250000

The idea behind Daisy, is that we wanted our domain users to be writing rules, but not code. Rather than using some high-powered rules engine, we wanted something exceptionally light-weight. We also wanted our domain experts to be inventive, and invent rules as they needed them. This is exactly what Daisy does.

Domain experts can write any rule they want, as long as it complies with the basic Boolean algebra syntax. Then, developers come along and write implementations to any rules that haven’t yet been implemented. In this way, it’s just like gherkin, but for business rules. Take a look at the code on GitHub or download the Nuget package.

]]>http://blogs.ancestry.com/techroots/announcing-daisy/feed/0Open Source Developmenthttp://blogs.ancestry.com/techroots/open-source-development/ http://blogs.ancestry.com/techroots/open-source-development/#commentsTue, 26 Nov 2013 09:27:23 +0000Liam Molloyhttp://blogs.ancestry.com/techroots/?p=1568At Ancestry.com, we have many developers whom contribute to open source projects. Today I want to talk about an open source project I have been involved in, along with one of my team members Shane Burke. As contributing to open source projects is not directly related to the day to day development work at Ancestry,… Read more

]]>At Ancestry.com, we have many developers whom contribute to open source projects. Today I want to talk about an open source project I have been involved in, along with one of my team members Shane Burke. As contributing to open source projects is not directly related to the day to day development work at Ancestry, it provides an opportunity to give back to the community and allows developers to think outside the box, as well as contribute to something they are passionate about. Both Shane and I work on this project in our spare time.

FooBar football is an opensource collectable card game (CCG) based upon football. It is inspired by Magic The Gathering, Football Manager, FIFA Ultimate team, Match Atax and a host of other games.

Over the next few months, I plan to write a series on what the game is and how it is being implemented. Here is a quick overview:

GitHub used for version control

AppHarbor used for hosting

C# used for match engine development

Angular JS used for client development

GitHub

For version control, we use GitHub. GitHub is a web-based hosting service for software development projects that use the Git revision control system. GitHub offers free accounts for open source projects. In our day-to-day environment Shane and I use Microsoft’s team foundation server for version control so contributing to this project has allowed us to learn something new.

You can check out the current state of the project on our GitHub page:

]]>http://blogs.ancestry.com/techroots/open-source-development/feed/0Ancestry.com Great Summer Experience for San Francisco Internshttp://blogs.ancestry.com/techroots/ancestry-com-great-summer-experience-for-sf-interns/ http://blogs.ancestry.com/techroots/ancestry-com-great-summer-experience-for-sf-interns/#commentsWed, 14 Aug 2013 21:19:43 +0000Alex Greenspanhttp://blogs.ancestry.com/techroots/?p=1029Picture: May 2013. Fog ever-present on the San Francisco skyline. College students sweating through their finals. And then, a light in the distance. Green, like the light hovering out past Gatsby’s dock. But this green is actually the Ancestry.com leaf, representing the new family that awaits four eager UC Berkeley students once they embark on… Read more

]]>Picture: May 2013. Fog ever-present on the San Francisco skyline. College students sweating through their finals.

And then, a light in the distance. Green, like the light hovering out past Gatsby’s dock. But this green is actually the Ancestry.com leaf, representing the new family that awaits four eager UC Berkeley students once they embark on their technical summer internships at the San Francisco office of Ancestry.com.

Throughout their summer, many trials and tribulations met them along their journey to Agile enlightenment (aka, figuring out how to get to all the conference rooms in the office). Here are a few highlights from their experience:

For FedEx Day, the SF interns joined to form a team and develop an application named “AnTESTry” in which Ancestry users’ knowledge could be tested about their own family trees. They were able to stay up collectively for over 24 hours in order to make the deadline and showcase their project in front of both the San Francisco and Provo offices. Though they did not win, they learned a lot, including MySQL, JSP, and CSS3 transitions.

One exciting event that took place outside of the office included a visit to Google for the monthly AngularJS Meetup. Two of the interns accompanied some full-time employees and had a blast eating free food, learning about AngularFire, and asking questions to get t-shirts.

Towards the end of the summer, the SF interns organized a back-to-school charity event and ordered school supplies for 160 sixth grade students at Visitacion Valley Middle School. They also invited the office to join in a packaging party to write personalized notes to the students and assemble the packages. Over 30 employees attended and a great time was had by all.

And now, what you’ve all been waiting for…introducing THE WINTERNS:

Tiffany!

Tiffany Hwu was a part of the Data Services team. Over the summer, she worked on use cases for different sources of data, and investigated RHadoop. She is currently entering her senior year at the University of California, Berkeley, double majoring in Computer Science and Cognitive Science. In her spare time, she plays piano and folds origami.

Tiffany! (THE OTHER TIFFANY)

Tiffany Ng was an intern for the Dispatch team. After learning a ton of new frameworks and languages such as C#, ASP.NET, AngularJS, and many others, she was able to help the team build an Admin User Interface for the new internal targeting service that Dispatch was in the midst of creating. In addition to helping with the UI, she wrote unit tests for the C# code and researched the Jasmine testing framework. Tiffany will be going into her fourth year at UC Berkeley, majoring in Computer Science. When she’s not coding, she likes to have fun food adventures and discover new music.

Alex!

Alex Greenspan was an intern for the Guidance team. She created a prototype for an easier way to explore record hints on the All Hints page, which will be implemented on the live Ancestry.com site. Among her other roles, she worked on unit tests and automation tests, as well as conducted research into the algorithm of the relationship calculator that can be seen on the upcoming side panel feature for Trees. Alex is going to be a third-year Computer Science and Cognitive Science double major at UC Berkeley in the fall. She’s taking baby steps toward saving the world. This involves a lot of photography and “creative brainstorming,” aka, post-it notes.

And Jake!

Jake Lerner was an intern for the DNA-Backend team. Rather than working on a specific “intern project,” Jake worked as a normal member of the team, improving the efficiency and logging capabilities of Ancestry’s DNA-processing pipeline. He’s going into his third year at the University of California, Berkeley where he is pursuing a double major in Computer Science and Political Science. When he’s not having a great time at Ancestry, he does research on the Political Economy of the Asia Pacific, cooks, reads science fiction, and climbs the occasional tree.

Overall, these four interns would unanimously agree that Ancestry.com is a great place for an internship. The atmosphere is friendly, the projects are interesting and challenging, the catered lunches on Mondays are delicious, and the surprises (such as ice cream trucks and bingo night) never cease to feel rewarding after a long week of work. If you know of anyone looking for an internship, point them to Ancestry.com…and not just to the website in their browser. But that too, because we’re sure their family history is worth discovering.

]]>http://blogs.ancestry.com/techroots/ancestry-com-great-summer-experience-for-sf-interns/feed/1Creating Random Data for Testinghttp://blogs.ancestry.com/techroots/creating-random-data-for-testing/ http://blogs.ancestry.com/techroots/creating-random-data-for-testing/#commentsWed, 12 Jun 2013 14:21:29 +0000Andershttp://blogs.ancestry.com/techroots/?p=80In my experience, tests that emulate real-world usage and use real-world data, find more relevant bugs, convey intent more clearly, and exercise the system under test more thoroughly than tests that do not. Consider testing a cab service to assert that a given vehicle arrives at its destination: cabService.SendVehicleToDestination(vehicle, destination); Assert.AreEqual(vehicle.Location, destination); Now, it shouldn’t… Read more

]]>In my experience, tests that emulate real-world usage and use real-world data, find more relevant bugs, convey intent more clearly, and exercise the system under test more thoroughly than tests that do not. Consider testing a cab service to assert that a given vehicle arrives at its destination:

Now, it shouldn’t particularly matter what vehicle is sent, or to where it is sent, but it’s often valuable to provide objects to methods that looks like the kind of objects an actual consumer of the service would send. Enter Randomator, a tool I created to use when scaffolding test objects:

This is, of course, a rather contrived example. We don’t have a cab service at Ancestry.com, fun as that would be, but as a family history company, we do have a lot person, event, location, and relationship objects for which this tool proves useful. Randomator has seen extensive use, enough so that I’ve ported it from C# to JavaScript and Java and made it available to the public under the MIT license. Check it out at: https://github.com/Ancestry/Testing-Utilities

]]>http://blogs.ancestry.com/techroots/creating-random-data-for-testing/feed/0Testing, Code Coverage, and Other Ways You Could Be Wasting Timehttp://blogs.ancestry.com/techroots/testing-code-coverage-and-other-time-wasters/ http://blogs.ancestry.com/techroots/testing-code-coverage-and-other-time-wasters/#commentsWed, 22 May 2013 14:45:43 +0000Andershttp://blogs.ancestry.com/techroots/?p=292I’ll be the first to say that testing and code metrics can improve software quality and increase productivity, but an overzealous application of either could incur a heavy cost.Tests are code, code is overhead, and while some overhead is necessary and even advisable, overhead is debt and should be minimized whenever possible.There is no perfect… Read more

]]>I’ll be the first to say that testing and code metrics can improve software quality and increase productivity, but an overzealous application of either could incur a heavy cost.

Tests are code, code is overhead, and while some overhead is necessary and even advisable, overhead is debt and should be minimized whenever possible.

There is no perfect product, bugs will be deployed to live systems (no matter how many tests and quality checks are in place), and for the most part, customers not only tolerate this, they expect it to one degree or another. What will really win over clients is how fast bug fixes are delivered.

Time is often better spent refactoring existing code, and improving feedback and logging systems. Making code testable will make it far easier to diagnose problems and expedite repairs. Tests should be put in place to prevent the same bug from surfacing more than once, but more time should be spent improving existing code quality than extending a safety net of tests to compensate for poor code quality.

I hear the term “code coverage” tossed around as if it were a panacea, it’s like the thought is, “if we improve our code coverage, it will solve all our quality problems”. Code coverage is useful tool, but certainly not the most important measure of quality. Depending on the code, 30% code coverage could be plenty, or in another case, 70% might not be enough – and keep in mind, that even if a particular piece of code is 100% covered, it may still need more testing. Code coverage is only one metric, and at best it’s an incomplete indicator of test coverage. All too often, a rule that requires 80% code coverage encourages testing of things like:

This is not testing the “Foo” class or business logic, and while it does increase code coverage, ultimately all that’s being tested is the C# language itself. This is not a good use of time since you can trust the language to do its job correctly.

Testing, code coverage, static code analysis, and related tools and practices, while important, have lower value than improving code quality and testability. Ultimately, no amount of success in testing can compensate for failure in code quality.