Podcast Episode 7 – The Art of the Possible – Moving a Mission-Critical System to the Cloud

Steven Borg: Today I talked with Brandon Ekberg, the Senior Vice President of Operations at Innovative Systems. And our talk ranged from the history of Innovative Systems. A 50-year-old company who’s had to pivot many times. And Brandon gives some tips and tricks on how to effectively pivot the organization, both from an organizational standpoint and from a technical standpoint. He also talked about how to move to the cloud and some of the things that you can expect when moving to the cloud. For instance, they found a 30% cost savings after moving to Azure. It’s a very interesting podcast, and Brandon’s an interesting character well worth listening to. So without further ado, onto the podcast.

Steven Borg: I’m here with Brandon Ekberg, the Senior Vice President of Operations for Innovative Systems. Brandon, can you give us a little history of yourself?

Brandon Ekberg: Sure. I went to Carnegie Mellon a long time ago, majored in mechanical engineering. Spent some time in the steel industry and really was wowed with some of the automation I saw there, went back to school, got a master’s in digital control systems and really been in software ever since. Worked for a company called Cimflex Teknowledge that did factory floor information systems. Then went to work for Rockwell Automation. They had a software-only division called Rockwell Software. Spent quite a few years there, in fact, helped set that up initially. I went to Eaton, helped them set up a software and communications group, and I joined Innovative Systems about three years ago.

Steven Borg: That is a wide-ranging history. That’s some interesting background. I’m guessing that some of that contributed to the history of Innovative Systems or ties in very cleanly because we’ve talked about that history, and I’d like to start the podcast, if we could, talking about Innovative Systems from its inception and some of the pivots you’ve made all the way through. Could you give a history of Innovative Systems?

Brandon Ekberg: Sure. The company is 50 years old this year. So it’s been around for quite a while. It was started by Bob Colonna, he is our founder, he’s still our Founder and CEO, very active in the operation of the company. Bob involved in really analyzing voter registration lists and helping in the different parties. I have to say I don’t know which side of the aisle he was on but helped different parties identify how to go after and advertise to voters and out of that, really develop some deep insight into how do you deal with lists of names? It sounds funny but you think about the different ways names are spelled, and sometimes it’s Mr. And Mrs. on the same line and you need to split those out into separate things and addresses. All the different nuances. So he took that core understanding, built a business that initially was going after cleansing mailing lists. You know? Develop some technology. For example, to add zip codes when they were coming out. And then expanded into what’s called householding where you would identify a group of people that were actually the same household, and instead of sending them six catalogs, you would send them one, which were greatly, you know, saved companies doing mailings a significant amount of money.

Brandon Ekberg: From that technology, again, that core engine keeps getting smarter and smarter of identifying what is a record of a person, you know? And how do you deal with that from a data point of view? Then became a major player in a burgeoning market called Data Quality, working mostly with financial institutions. And where they really gained a solid foothold was, in those days, you were building special systems, one for checking, one for credit cards, one for savings account, and they were all separate, right? Because software was really just moving into the business world.

Brandon Ekberg: What you didn’t have in those systems was a view of your customers. So, like, which customers have checking accounts, which have savings, which have credit cards? So, they develop that, again, using that name identifier technology, they developed the ability to build customer information files where they were the central repositories used by banks to truly understand who were their customers and what the solutions or what products that they bought from the bank.

Steven Borg: That’s an interesting pivot because it’s a small pivot from the first part, but cleaning mailing lists and things like that, understanding the difference between someone spelling out the word avenue and A-V-E, that’s important. It cleans everything up, it gets the political parties to send things out correctly or anybody else and deduping, that kind of understanding. But now you’ve kind of broadened that technology. That’s a pretty big step now when you’re talking financial institutions as they’re starting to merge those things together. That had to have been a crazy time.

Brandon Ekberg: That was the first of their big pivots. And it really did bring them into a new marketplace. And it’s really where the company took off. As you can imagine, that became a fantastic tool for the financial institution marketing departments. They could now improve their cross-sell, right? Which you go send mailers to, or even call or whatever you’re doing, sell credit card to people who had checking accounts and savings accounts. Sell them a car loan. All the things that we see now. A lot of modern banking platforms today are built on that common information or that customer information centered model. So the idea of a separate system isn’t quite as prevalent anymore. But Innovative Systems was there at the beginning and really helped drive that part, that offering.

Steven Borg: And from that then, like you mentioned, banks started to integrate all of that, and that kind of integrated customer-centric view has kind of seeped in and expanded through the marketplace. It’s kind of a default. Where did Innovative Systems go next?

Brandon Ekberg: Yeah. Well, it was interesting. We had this great technology where we could very accurately identify a person, you know, from multiple sources, multiple records and say, “Oh, this is actually the same person,” even though a name or some piece of information was not quite right. Actually, when the Patriot came out in 2001, it really opened a whole new market up to us. And that was that the financial institutions were now legally obligated to scour all of their list of customers, they were required to check when they onboarded a new customer and ensure that you weren’t dealing with any known terrorist or any known money launderer.

Brandon Ekberg: And that capability, our capability to match names really fit perfectly ’cause now we could take the client records from the financial institution and compare them against all these lists that are available from around the world. So, we actually provide our customers, I think we have about 40 different lists that we pull from, the United Nations, from the US government, from the British Treasury. So all these different institutions put out their list of people that are actually sanctioned. So this literally means a law has been passed and you’re not allowed to do business with those people.

Brandon Ekberg: We provide those because those lists are publicly available information and we pull them from all the different sources, clean them up and optimize it so that we can scan against them quickly. But where it really even grew is that companies like Thomson Reuters and Dow Jones and different companies out there have these very well researched lists that are called politically exposed people. And they’re PEPs lists, and those lists are sold to financial institutions. And we also use those lists. So we now provide a service to financial institutions where whether they’re onboarding a customer or whether they’re doing their obligatory scan of their installed base, most banks do it every night, we can combine that, the information of the bad guys against the clients and identify any people that they potentially should not be doing business with. And that’s really been the booming part of our business over the last… It’s really taken off since about 2005.

Steven Borg: That is a really interesting history. And coming from a case of disruption or change inside of an organization, that’s two relatively major pivots that the organization has gone through.

Steven Borg: I want to dive into this with you a little bit, Brandon. How did Innovative Systems make those transitions? Those seem like big organizationally wrenching transitions.

Brandon Ekberg: You know, they’re not as big as they seem in the sense that we were listening to our customers. So, we were initially thinking about the mailing lists. Some of our customers that we were working with were banks and financial institutions doing mailings, and they were telling us about some of their problems. And we said, “Wow, you know, what we do to your mailing lists, we can do to your internal systems.” And then you continue going forward with that, and again, we’re servicing the financial community. And we’re able to take a look at what their needs are, we’re listening to them every day. And the cool thing is, that core engine that started out in an IBM mainframe, it’s gone through a few technology life cycles, so it’s gone from IBM mainframes and then we did open systems. You know, now we’re primarily a Windows technology stack. But that core technology of matching those names is still core to what we do.

Brandon Ekberg: In fact, one of the things we do better than really anybody in the marketplace is how fast and efficient we are. We can scan 20, 30 million records overnight without a problem at all. We do it on our hosted systems. We just passed, for the last 12 months, we’ve done 70 billion transactions. So a transaction is looking up a name. So we can really, really scale. And some of it, where it comes from is we carry forward some really efficient C code that was ported from assembler on an IBM mainframe. And that’s when memory really mattered and your code better be really, really efficient. So, that’s some of our core capability today.

Steven Borg: That’s interesting. I totally get it. In so many organizations, there’s that kind of legacy code that makes its way all the way through. Is there a desire to rewrite that? How has that made that transition all the way through?

Brandon Ekberg: We continue, obviously, to optimize it. We’ve got a team of great developers. We just hired two young people, and C-code typically isn’t the first language of folks coming out of the universities today. But we looked very hard and found some great people, and there are parts of our engine that we still do use C from very, very, you know, that optimization part of our system.

Brandon Ekberg: A lot of our system’s written in more of the Microsoft stack. .NET code for some of the front end stuff, we do a lot of our web stuff is done in Ajax, and we’re in the process of laying out an XGen UI going to React.

Steven Borg: Again, interesting. So, I want to come back because you mentioned one thing on that pivot. And it just kind of seemed really important to pull out. And I said those pivots, those are kind of transitional pivots that are very difficult for organizations to often do. And you said, “Oh no. We were listening to our customers.” And I just want to pop that out because that seems like such an amazing piece of advice. Pretty obvious in a lot of cases but you were hand in hand with those customers. You have anything else, any other kind of top line things that Innovative Systems did in order to make those changes and to follow the market and your customers effectively?

Brandon Ekberg: I think it’s a combination of listening to your customers. I think we also stayed within an industry, so we weren’t bouncing all around. So we understood the language. We understood the needs. And we also understood our own capabilities. So we understood what our core platform could do, what our core competency is. We define our core competency as being able to manage large datasets of records of people or companies. And by staying the course around those things, we were able to pivot fast and be effective at it.

Steven Borg: So let’s shift gears then and talk a little bit about some of your technology shifts as well. So, we got kind of the culture or organizational agility that you’ve been able to do by following your customers, staying in the groove, really paying attention to where your capabilities are. What technological shifts have gone along with that? You’ve carried along some of the C code, but what other changes have occurred? You mentioned mainframe all the way through to today. What shifts have you made?

Brandon Ekberg: You can just follow technology, right? So on the mainframe, it is all batch processing, executable UI really doesn’t matter. In fact, in a lot of cases, the UI doesn’t even exist. You know, then as you go forward, you get into more of a classic client-server architecture where you had a client so you could configure things. You brought more, you know, or maybe some of the batch stuff was all configured in some sort of .DAT file with the… we’re all familiar with those kind of things. INI files, we all remember those. Reg keys, right? In the Windows world?

Brandon Ekberg: Great way to get yourself in trouble. So, we moved from that to a client-server architecture. Like I said, it’s our batch processes, the really high-speed stuff is still a core executable batch process. And then we now front that system. Probably the big change we’ve made in the last few years is we now have a web application. So we have a .NET web application that is both the user front end, so it replaces that client, the client we used to have, so now it’s a pure browser.

Brandon Ekberg: And then more and more of our customers are using web interfaces or, you know, web services to work with us where five years ago, most of our customers would hand us a csv file or an html file or an XML file, I’m sorry. Where today, a much larger percentage of customers are actually making web service calls to, interfaced us and we’ve become a little more of a back-end process. And that’s also brought us, where we’ve done more hosted activity. So, it used to be we did everything on prem. Over the last probably five or six years, we’ve seen a rapid adoption of especially financial institutions willing to put the kind of information that we have, right? Because it’s critical. It’s a customer name and could be an account number, depending on what they’re using to do the matching, and they’re willing to hand it to us in a hosted environment.

Steven Borg: I didn’t catch that in some of our earlier discussions. Brandon, that is a massive amount of trust that’s being extended by those financial institutions to you. How do you manage that relationship and that security? How does that work?

Brandon Ekberg: We have to prove to the financial institutions that we know how to operate a data center. We don’t operate the, or initially, we really basically bought bare metal VMs in some of the regional data centers like in Germany and Singapore and Latin America and things like that. We did have a colo here in Pittsburgh that we actually managed the hardware ourselves. But we did, we had to prove to the banks that we could manage their information. I mean, we have a separate hosted team, only that team is allowed to get access to the back-end systems where you can get access to the data. Even our professional services people, we segment their access. They can get access through the web UI which has, there’s a limited things you can do through the web UI. But any of the back-end tasks are relegated to a much smaller group of people.

Brandon Ekberg: In fact, as you know, as we met 10th Magnitude, it was all about trying to get off of our own virtual machines and get onto Azure. And one of the great things there is that if you take that trust center URL and send it to a customer that answers such a huge number of questions they have about how do you manage your center and do you have all the right certifications, so that’s really helped a lot.

Steven Borg: That’s good to hear. Well, especially from a 10th Magnitude perspective ’cause we bet the cloud on Azure. We love the Azure stacks. It’s good to hear that that helps people feel comfortable as they move over with you.

Brandon Ekberg: Our experience over the last year running on Azure, or Azure, is a great bet you guys made.

Steven Borg: How do you pronounce it? ‘Cause I keep pronouncing, I heard you pronounce it twice.

Brandon Ekberg: I always say Azure. I guess I go back and forth, Azure, Azure.

Steven Borg: I just thought that was a fun question because I hear Azure and I hear Azure, and those are both in the same conference at Microsoft. So I’m not sure there’s a correct pronunciation or not.

Brandon Ekberg: That’s interesting. Well, isn’t it a French word for a form of blue?

Steven Borg: Ah, yes.

Brandon Ekberg: I think it is. So there’s probably an official French way to pronounce it. I’m not positive, but I think that’s true.

Steven Borg: That might be even harder for me to pronounce that.

Brandon Ekberg: Yeah, yeah.

Steven Borg: So, let’s shift back. You made that transition into the cloud. What are some of the lessons learned from that transition? Because mainframe to client server’s a very large transition, client-server to that web front end, again, a fairly large transition. And then no matter what, when you’re moving from a non-premises thing to the cloud, what kind of difficulties did you run into or what lessons learned do you have?

Brandon Ekberg: So, this is where 10th Magnitude was a huge help. We met you guys, we were introduced by Microsoft, and they recommended we talk to you guys about planning a migration, understanding what the issues were. So you helped us understand. I mean, initially, it seemed like, “Well, you know, it’s a Windows operating system, and it’s SQL server in the cloud.” You know, there’s a few nuances there. SQL server in the cloud works just a little different about how it manages connections, right? ‘Cause it’s a shared resource. The things you learn and you guys and 10th Magnitude really helped us identify those. There were some things in our product, especially around managing connections to the SQL server instance, that we had to work through and make sure we did correctly so that it would operate in the Azure shared services environment.

Brandon Ekberg: But after that, once we got through that issue, then the next big one was actually the migration of the data. I mean, we had five centers around the world, and we had to migrate those. Our largest one was the United States, and we had a couple terabytes of data. And our system is a mission-critical system, so we have to be alive, right? We have to be up and running to support these banks and these financial institutions. So we had to really come up with a way to migrate or we could do it in a very short weekend, and by short weekend I mean we had to really do the migration from Friday night into Saturday so that we could have Sunday to confirm everything operated correctly and do the testing.

Brandon Ekberg: And moving that data from, for example, our colo in Pittsburgh up to the Azure East site, which I don’t even know where it is, probably Philadelphia somewhere? I don’t know. That took a few tries. We did it a few different ways, and then finally, 10th Magnitude introduced us to a team at Microsoft that was building a brand new migration tool called Data Management Services, DMS, and we’re actually one of the beta participants in that, and that proved to be very, very effective, and that was the tool we used to do that migration, get up and running. The US site was our first site, and that went very well. Over a weekend, we shut down, migrated, came up Sunday afternoon, everything worked perfect.

Steven Borg: That’s good to hear. How long ago was that? I’m just curious on that tool.

Brandon Ekberg: So, let’s see. We migrated in December, we went live on Azure in the US in December of ’17. So, a year ago. And we did all the… Yeah, we really started working heavily with you guys probably starting in September? Maybe a little earlier than that? Maybe August? Really planning things out. We did multiple dry runs, as you can imagine, we wanted to know that it was gonna work. That was through most of that fall and the winter and then we went live with our first site. We did US in December, we did Canada in January, we did Brazil. Actually, we were in the Dominican Republic I think is where we were. And we moved from a set of virtual machines there to the Brazil site in April. And then we moved Germany in June. And we’ll do our last move in Singapore in March of ’19, and that really has more to do with the fact that we have a contract with a provider there that doesn’t expire til ’19, until the March ’19, and we didn’t want to pay double.

Steven Borg: Totally makes sense. Hey, I want to shift gears a bit. You mentioned you get millions of these hits where people are finding out, is the person we have asking to deposit money into our account, maybe doing a check more than 10k or something like that, are they on some kind of money laundering watch list or those sorts of things. How do you handle things where you can’t quite tell? There’s duplicate identities. There’s some times when I feel like I might be on a watch list somewhere where I’m trying to go through TSA. How do I dedupe that? Find out-

Brandon Ekberg: Yeah, that’s a good question. I mean, that’s kind of core to what we do. So, in the industry, in our industry, the name screening or anti-money laundering, your ability to manage false positives is really critical. So one of the things that we have is that that sophisticated engine that can identify a record and say, “This is actually the same person,” and deal with the nuances of slight spelling differences and even alias and all different kinds of things we’ve built over the years, we have a very unique algorithm for doing that matching. And it’s highly tunable, so we can work with customers to tune it.

Brandon Ekberg: So, we typically get that false positive, or what we call the match rate, down to about 3% of what’s submitted. Which is much better than most of our competitors, and that’s really one of other areas we excel in. But still, you’re gonna get into that 3% world, right? So if you’re a bank and you got 10 or 20 million customers, those, you still have some work to do. You have to go through and review those cases. So some banks may have a team of 20, 30 people that review those, the outputs of these systems, do the extra research needed to try to decide, so maybe the system said, here’s the client name, which meant they came from the bank side, and here’s the three records that we think match that person.

Brandon Ekberg: And a human being jumps in and does some research and tries to decide whether that really is, and if they clear a case, as it’s called, they typically have to make notes that are available to an auditor if you ever want to be able to prove why did you make a decision that this wasn’t the bad person? And so there’s a lot of stuff there. In fact, that’s one of the big areas we’re doing research in now is trying to figure out how can we apply artificial intelligence or machine learning there whereas our system watches what the human beings do to clear these cases, can we further fine-tune our algorithm to improve our ability so that we don’t send them cases that weren’t really matches? And the thing you have to be careful with is you don’t want to get to the point where you actually miss somebody who really needed to be identified. And that’s kind of the risk. It’s called a false negative. To try to avoid those at all cost.

Steven Borg: Yeah. I guess that’s the case in most machine learning algorithms when we’re tuning or you’re doing some kind of, well, any statistical work. You try to balance the false positive rate with the false negative rate in some way, shape, or form. In medicine, you want a false positive, right? You’d rather tell somebody they have some disease and then back off and say, “Oops. Further testing shows you’re clear.”

Brandon Ekberg: Yeah.

Steven Borg: As opposed to-

Brandon Ekberg: To missing it.

Steven Borg: …getting it wrong.

Brandon Ekberg: It’s a great analogy. It’s the same thing. And the tradeoffs between false positives and false negatives is the risk profile of the institution.

Steven Borg: And that’s one of those tunable parameters then? That customer can say, “Meh, give us a little more leeway so we have maybe 1% or 2% on the false positive side.”

Brandon Ekberg: Yeah. Look, yeah. And we are highly, highly tunable. One of the unique things we do is that for… And typically in a compliance application, which is looking for terrorist and money launderers, things like that, you have about 10 fields you’re willing to deal with. Being first name, last name, date of birth, country of origin, different information like that. And one of the things that we do very uniquely is we match, we literally give you this kind of cool little color graphics that tells you when we identify a match is to whether each of the 10 fields is an exact match, you know, which we color code green, a close match, which we color code as yellow. We’ve probably got about, I don’t know, 30 or 40 different rules you can apply to say what does close mean? You know?

Brandon Ekberg: And then there’s blank which just means the records are blank, so we’re not even gonna make an evaluation. And then there’s a not a match, so we’ve got this little color code that’s green, yellow, red, and gray. And you kind of look at that color code bar when you look at a record, and it gives you a really great visual indicator as to whether it’s a match or not. And then behind the scenes, when we talk about that fine-tuning that algorithm, you can imagine we can literally fine tune at the field level. Exactly how those match and how those work where a lot of companies in this industry, really, what they do is they just say, “I want to do an 85% match,” and then there’s an algorithm that runs through all the different fields and comes up with some numerical weighting and decides what 85% means.

Steven Borg: Yeah. This problem, it sounds so obviously simple, right? Oh, we just need to match up and see if these two people are the same. But I have been in organizations, state organization here in Washington that’s just been struggling with trying to do this for years. And just dedupe so that they can understand who belongs to which part and who is each individual for child support reasons all the way through driver’s license and who are these folks who’s voting? And matching that all up is a brutal, hard problem.

Brandon Ekberg: Absolutely. Yup.

Steven Borg: So, it’s kudos to solving that problem or getting a 97% rate where you can have full confidence, and that’s tuned to avoid false negatives highly.

Steven Borg: What advice would you give our listeners in making some of these shifts? Moving to the cloud, you mentioned some of the things you did, but do you have any tips and tricks? Things that they might want to watch out for? You mentioned SQL doesn’t run the same or those sorts of things. Anything else?

Brandon Ekberg: I would say the biggest tip I could give is that as you go into something new, finding a partner that you can trust that can help you on that path is so critical. I mean, that’s really what helps. Having a guy like Mark Johnson from 10th Magnitude go through our architecture, understand how our products work, understand how we use the different Microsoft platform utilities, he was able to help us understand where the pitfalls are gonna show up. We would do testing to try to see if they really did exist, sometimes they did, sometimes they didn’t. It really was having that knowledgeable person that could help us out was really, really key.

Steven Borg: That seems legit. And also having such a good team behind you. I can’t tell you how often I’ve gotten calls from members of 10th Magnitude team saying, “You’ve got to get Innovative Systems on the phone. You’ve got to talk with them. They are doing amazing things.” And they kept talking about how, well, frankly, how innovative you are. I guess it’s baked into your name. So, it’s been fun to explore some of those things like the customer-centric focus and knowing your limits. And the other things that you’ve done does so effectively, kind of roll with the punches as industries have consolidated their customer-centric stuff in-house and then making your many, many pivots.

Steven Borg: This has been a really interesting conversation, and as we wrap, is there any last things you’d like to say, Brandon?

Brandon Ekberg: No, it’s been great. It’s been a fun road. The benefits we gained of working with 10th Magnitude, moving to Azure were just awesome. I mean, we gained significant savings, we’ll probably see over 30% cost reduction, our hosted costs as we head into 2019, and that’s really great. And we’ve improved the robustness of our offerings. So that’s all been great. It’s been a fun trip and loved working with 10th Magnitude.

Steven Borg: Thank you very much. And one last thing before we go. Where do people find more about Innovative Systems if they’re interested in working there or they’re interested in acquiring your software?

Brandon Ekberg: So InnovativeSystems.com is our home website. And we talk about our different product offerings up there, both in data quality and compliance. We usually always have multiple jobs, we’ve been growing fairly rapidly over the last few years. So we’re always looking for good people and we have offices in London, Pittsburgh, and Mexico City, so if you’re in any of those regions, look us up. And we actually have people, we hired a gentleman in Riyadh, Saudi Arabia, because we’ve got a good collection of customers in that region. And the ability to support both the shifted week and the time zone, that was really important for us to get local resources there, but our prime offices are London, Pittsburgh, and Mexico City. Pittsburgh is our corporate headquarters. Best city in the world.

Steven Borg: It’s a neat place. I’ve been there a few times and have enjoyed it each and every time.

Brandon Ekberg: I love it.

Steven Borg: Brandon, thank you very much.

Brandon Ekberg: My pleasure, thanks for the opportunity. It was a fun chat.

Steven Borg: Thanks for listening to the Art of Digital Disruption. At 10th Magnitude, we’re proud to create the path for organizations to stay competitive and disrupt their industries. And for more information on innovation and how you can disrupt your industry, visit 10thMagnitude.com/agilityquadrant and download our latest Whitepaper. Thanks for listening.