Snippets tagged regex

This function produces safe URLs for Web requests or URI construction. It forces the pattern:
http:// + www. OR subdomain. + domain + absolute path.
When used in an application that takes URLs as input, the user would be able to type "example.com" instead of "http://example.com" or "http://www.example.com". It also supports domains like google.co.uk or google.com.au.

This snippet features an F# Web crawler that i'm already using in 2 applications (slightly modified). It's based on a scalable network of communicating agents that follow URLs extracted from HTML pages until reaching the specified limit.