Buck Woody : Conferences, SQL Server, PASShttp://sqlblog.com/blogs/buck_woody/archive/tags/Conferences/SQL+Server/PASS/default.aspxTags: Conferences, SQL Server, PASSenCommunityServer 2.1 SP2 (Build: 61129.1)Big Data and the Cloud - More Hype or a Real Workload?http://sqlblog.com/blogs/buck_woody/archive/2011/10/18/big-data-and-the-cloud-more-hype-or-a-real-workload.aspxTue, 18 Oct 2011 13:57:36 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:39156BuckWoody0http://sqlblog.com/blogs/buck_woody/comments/39156.aspxhttp://sqlblog.com/blogs/buck_woody/commentrss.aspx?PostID=39156<p>Last week Microsoft announced several new offerings for “Big Data” - and since I’m a stickler for definitions, I wanted to make sure I understood what that really means. What is “Big Data”? What size hard drive is that? After all, my laptop has 1TB of storage - is my laptop “Big Data”?</p> <p>There are actually a few definitions for this term, most notably those involving the <a href="http://nosql.mypopescu.com/post/9621746531/a-definition-of-big-data" target="_blank">“Four V’s” Volume, Velocity, Variety and Variability</a>. Others <a href="http://nosql.mypopescu.com/post/10120087314/big-data-and-the-4-vs-volume-velocity-variety" target="_blank">disagree with this</a> definition. I tend to try and get things into their simplest form, so I’m using this definition for myself:</p> <p align="center"><font color="#c0504d" size="3">Big data is defined as a <em>large set </em>of <em>computationally expensive </em>data that is <em>worked on simultaneously</em>.</font> </p> <p>Let me flesh that out a&#160; little. To be sure, “Big Data” has a larger size than say a few megabytes. The reason this is important is that it takes special hardware to be able to move large sets of data around, store it, process it and so on. (<font color="#c0504d">large set</font>)</p> <p>If you store a LOT of data, but only use a small portion of it at a time, that really isn’t super-hard to do. It’s mainly a storage issue at that point. But, if you do need to work with a large portion of the data at one time, then the memory, CPU and transfer components of the system have to adapt to be responsive - new ways to work with that data (game theory, knot-algorithms, map-reduce, etc.) need to be brought into play. (<font color="#c0504d">computationally expensive</font>)</p> <p>Once that data is loaded into the processing area (memory or whatever other mechanism is used) it must be worked on in parallel to come back in a reasonable time. You have two options here - you can scale the system up with more internal hardware (CPU’s, memory and so on) or you can scale it out to have multiple systems work on it at the same time using paradigms such as map/reduce and so on. Actually, when you lay this out in an architecture diagram, scale up or out doesn’t actually change the logical structure of the process - in scale out the network becomes the bus, and the nodes become more RAM and computing power. Of course, there are changes in code for how you stitch the workload back together. (<font color="#c0504d">worked on simultaneously</font>)</p> <p>So back to the original question. Is Big Data, as I have defined it here, a workload for Windows and SQL Azure? Absolutely! In fact, it’s probably one of the main workloads, and I believe it represents the latest, and perhaps also the earliest frontier of computing. Jim <a href="http://research.microsoft.com/en-us/um/people/gray/" target="_blank">Gray, a former researcher here at Microsoft and a hero of mine, was working on this very topic.</a> I believe as he did - all computing is simply an interface over data. </p> <p>Microsoft has multiple offerings on the topic of Big Data. In posts that follow from myself and my co-workers, we’ll explore when and where you use each one. Whether you are a data professional or a developer, this is the new frontier - <a href="http://www.straightpathsql.com/archives/2011/10/microsoft-loves-your-big-data/" target="_blank">don’t wait to educate yourself</a> on how to leverage Big Data for your organization. </p> <p><strong>Hadoop on Windows Azure and SQL Server&#160; </strong>- Microsoft’s <a href="http://www.hortonworks.com/the-whys-behind-the-microsoft-and-hortonworks-partnership/" target="_blank">partnership to include Hadoop workloads on Windows Azure</a> and <a href="http://www.microsoft.com/download/en/details.aspx?id=27584" target="_blank">SQL Server/Parallel Data Warehouse (PDW)</a></p> <p><strong>LINQ to HPC </strong>- Microsoft’s High-Performance Computing SKU of <a href="http://blogs.technet.com/b/windowshpc/archive/2011/05/20/dryad-becomes-linq-to-hpc.aspx" target="_blank">HPC is now in Azure</a></p> <p><strong>Windows Azure Table Storage </strong>- A <a href="http://msdn.microsoft.com/en-us/library/windowsazure/hh508997.aspx" target="_blank">key/value pair type storage with full partitioning</a> that is immediately consistent, able to handle huge loads of data and works with any REST-compatible language</p> <p>&#160;<strong>Other offerings </strong>- Including the new <a href="http://www.microsoft.com/en-us/sqlazurelabs/default.aspx" target="_blank">Data Explorer</a>, <a href="http://research.microsoft.com/en-us/news/headlines/daytona-071811.aspx" target="_blank">Project Daytona (with a Big Data Toolkit for Scientists and researchers)</a>, <a href="http://www.microsoft.com/sqlserver/en/us/future-editions/SQL-Server-2012-breakthrough-insight.aspx" target="_blank">Power View</a> and more. </p> <p>The era of Big Data is here. And you can use Windows and SQL Azure to bring it to your organization. </p><img src="http://sqlblog.com/aggbug.aspx?PostID=39156" width="1" height="1">DeveloperMicrosoftDBASQL ServerCareerSQL AzureConferencesPASSSQLServerDataData ProfessionalCloudAzureWindows AzureCloud ComputingStorageConceptsAzure Use CasesPolicy Based ManagementSQL Saturday 27 (Portland, Oregon)http://sqlblog.com/blogs/buck_woody/archive/2010/05/24/sql-saturday-27-portland-oregon.aspxMon, 24 May 2010 13:49:00 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:25473BuckWoody0http://sqlblog.com/blogs/buck_woody/comments/25473.aspxhttp://sqlblog.com/blogs/buck_woody/commentrss.aspx?PostID=25473<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;"><o:p></o:p></SPAN></P>
<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;">I’m sitting in the Seattle airport, waiting for my flight to Silicon Valley California for the SQL Server 2008 R2 Launch Event. By some quirk of nature, they are asking me to Emcee the event – but that’s another post entirely. <o:p></o:p></SPAN></P>
<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;"><o:p>&nbsp;</o:p></SPAN></P>
<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;">I’m reflecting on the SQL Saturday 27 event that was just held in Portland, Oregon this last Saturday. These are not Microsoft-sponsored events – it’s truly the community at work. Think of a big user-group meeting – I mean REALLY big – held in a central location, like at a college (as ours was) or some larger, inexpensive venue like that. Everyone there is volunteering – it’s my own money and time to drive several hours to a hotel for the night, feed myself and present. It’s their own time and money for the folks that organize the event – unless a vendor or two steps in to help. It’s their own time and money for the attendees to drive a long way, spend the night and their Saturday to listen to the speakers. Why do all this?<o:p></o:p></SPAN></P>
<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;"><o:p>&nbsp;</o:p></SPAN></P>
<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;">Because everybody benefits. Every speaker learns something new, meets new people, and reaches a new audience. Every volunteer does the same. And the attendees? Well, it’s pretty obvious what they get. A 7Am to 10PM extravaganza of knowledge from every corner of the product. In fact, this year the Portland group hooked up with the CodeCamp folks and held a combined event. We had over 850 people, and I had everyone from data professionals to developers in my sessions.<o:p></o:p></SPAN></P>
<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;"><o:p>&nbsp;</o:p></SPAN></P>
<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;">So I’ll take this opportunity to do two things: to say “thank you” to all of the folks who attended, from those who spoke to those who worked and those who came to listen, and to challenge you to attend the next SQL Saturday anywhere near you. You can find the list here: <A href="http://www.sqlsaturday.com/"><SPAN style="mso-bidi-font-family:'MS Shell Dlg 2';">http://www.sqlsaturday.com/</SPAN></A>. Don’t see anything in your area? Start one! The PASS folks have a package that will show you how. Sure, it’s a big job, but the key is to get as many people helping you as possible. Even if you have only a few dozen folks show up the first time, no worries. The first events I presented at had about 20 in the room. But not this week.<o:p></o:p></SPAN></P>
<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;"><o:p>&nbsp;</o:p></SPAN></P>
<P style="MARGIN:0in 0in 0pt;" class=MsoNormal><SPAN style="FONT-FAMILY:'MS Shell Dlg 2','sans-serif';COLOR:black;FONT-SIZE:9pt;">See you at the Launch Event if you’re near the San Francisco area tomorrow, and see you at the Redmond SQL Saturday and TechEd if not.<o:p></o:p></SPAN></P>
<P>&nbsp;</P><img src="http://sqlblog.com/aggbug.aspx?PostID=25473" width="1" height="1">SQL ServerCareerConferencesCustomer ContactPASS