Parsing a URL with SQL Server Functions

Problem

In my environment I have a few tables with thousands of rows that have URLs in particular columns. We are going through some changes in the environment and need to be able to parse out the data (base URL, file name, query parameters) from the specific columns then move the parsed data to another database design. Moving the data is no problem. I plan on using SQL Server 2005 Integration Services and I have a good sense of how to use that tool. By the way, all of the URLs are in varchar fields if that makes any difference. Can you provide any recommendations on how to parse that data?

Solution

Although there are probably a few different ways to approach this task, let's break it apart and see how we can code each one of these scenarios with T-SQL commands in SQL Server 2005. I assume you will be parsing the data from a single column to three new columns in a new database\table design. Since you are familiar with SQL Server 2005 Integration Services, I will not dig into those details and just focus on how to capture the data that you need. So let's take a look at each scenario and give an explanation on the T-SQL coding in SQL Server 2005.

Scenario 1 - Base URL

As an example, let's say the data you are working with looks like 'http://www.mssqltips.com/sqlservertip/1156/opportunities-with-sql-server-2005-express-edition/' and you need to capture the 'www.mssqltips.com' portion of the column. Below is an example of what that code could look like. This code is fairly straight forward with a single SUBSTRING command performing all of the heavy lifting. The REVERSE command is used to find the last slash so the entire directory structure is returned. The CHARINDEX command is used to determine the position of characters such as '//' in 'http://' and '/' in the URL directory path.

Scenario 2 - File Name

In this scenario let's capture file name which is the 'tip.asp' portion of the 'http://www.mssqltips.com/sqlservertip/1156/opportunities-with-sql-server-2005-express-edition/' URL. This code makes additional usage of the REVERSE and SUBSTRING commands, but the subtle portion of this code is positioning each portion of the SUBSTRING correctly. This is accomplished by using the '+1' in the starting position (second position) of the SUBSTRING command and using a '-1' in the length position (third position) of the SUBSTRING command. In this circumstance the CHARINDEX command is used to determine the position of the '?' and '/' characters. Here is an example of what that code could look like:

Scenario 3 - Query Parameters

In the final example capturing the query parameters is based on finding the correct position for the question mark '?' in the URL. With the 'http://www.mssqltips.com/sqlservertip/1156/opportunities-with-sql-server-2005-express-edition/' sample data, you would need to capture the 'tip=1156' portion of the column. Here is an example of what that code could look like by leveraging the REVERSE, CHARINDEX and SUBSTRING commands once again in a little different context:

Next Steps

Fire up SQL Server 2005 Management Studio, create the table, load the data and start running through this code. It is fairly simple, but could serve as a good first means to learn about manipulating strings.

The next time you need to parse out portions of a URL or another string, consider some of the techniques used to accomplish the task in this tip. The SUBSTRING function is typically heavily used and the REVERSE came in handy this time based on the needs. So get creative!

--Unless you iterate through a list of TLDs, you can't differentiate between subdomain.example.com and example.com.au --I decided against this because it's slower, and the TLD list requires maintenance