That was the scene at the inaugural Yahoo! Publicis Hackday at the Digitas office in downtown Manhattan last week, sponsored by the Yahoo! Developer Network. The highly competitive 24-hour hack event brought together top talent from Publicis agencies including developers, content strategists, UX, designers and project managers, all using Yahoo! technologies and platforms like YUI, YQL, Flickr and Yahoo! Finance to collaboratively rethink innovative digital ads, concepts, apps and tools for advertisers and consumers.

Innovation is best manufactured in a partnership environment and we believe this joint approach provides creative grounds for digital media and advertising development, engagement and reinvention. We caught up with Andrew Steiger, Digitas vice president & group director, digital experiences and technology, at the event for his thoughts:

We have a great partnership with Yahoo! to create best-in-class media

Today we kicked off the fourth annual Hadoop Summit at the Santa Clara Convention Center, bringing together some of the most influential thought leaders in the space, including Yahoo!, Facebook and IBM, to collaborate on Big Data issues and share experiences in building, managing and operating relevant real-world applications on Hadoop.

Yesterday, together with Benchmark Capital we announced the formation of Hortonworks. This new independent company was founded based on the Hadoop technology pioneered at Yahoo! and will be led by key architects and core contributors to the open source Apache Hadoop technology.

Originally launched in November 2007, the Yahoo! M45 academic research initiative provides universities the opportunity to conduct research otherwise impossible without the power and speed of Yahoo!'s supercomputing resource, which consists of approximately 4,000 processors.

Yahoo! is now expanding its supercomputing cluster to include four additional marquee universities: Stanford, the University of Washington, the University of Michigan at Ann Arbor, and Purdue. They join Carnegie Mellon University, Cornell University, the University of California at Berkeley, and the University of Massachusetts Amherst on the cluster, which brings a unique Internet-scale computing environment to academic researchers.

Hadoop, the open source technology at the epicenter of big data and cloud computing, is the core data-analysis technology used across Yahoo!. It is used by all universities participating in the M45 research initiative.

On Monday, November 8, 2010, the Federal Communications Commission will sponsor an Open Developer Day event at FCC Headquarters in Washington, DC, to promote collaboration between Web developers in the public and private sectors, in furtherance of FCC goals to further innovation in accessible technologies and foster citizen participation in open government.

This will be a public, single-day event that prioritizes accessibility goals, though other Web solutions are also of interest. The event will feature guest engineers from the Yahoo! Developer Network and Yahoo!s Accessibility team, and will have a component addressing the requirements and opportunities in the new Twenty-First Century Communications and Video Accessibility Act.

The event will run from 9:00 AM to 5:30 PM, with registration between 8:30 AM and 9:00 AM. There will be no charge to participate. Any developer is welcome to participate, and is expected to bring his or her own laptop computer.

Over on the Yahoo! Search blog, Rahul Hampole describes the new paid model for BOSS (Build Your Own Search Service): "Early in 2011, BOSS will transition to a paid model and we will round out the new version with monetization options and advanced security access (OAuth). The new BOSS V2 platform will initially offer a cost-per-query model. In addition, we plan to offer a nominally-priced version of BOSS for academia and non-profit organizations."

Oozie v1 is a PDL workflow server engine for Hadoop that enables creating workflow jobs composed of several map-reduce jobs, Pig jobs, HDFS operations, and Java processes. Workflow jobs are monitored as single unit via Web services, a Java API, and/or a Web console. Oozie v1 is in production in Yahoo!, and it has become the standard way for doing computations that require the coordination of multiple Hadoop/Pig jobs.

Media Production by BAYCAT, a non-profit community media producer that educates and employs underserved youth and adults in the digital media arts.

Existing best practices for MapReduce graph algorithms have significant shortcomings that limit performance, especially with respect to partitioning, serializing, and distributing the graph. Jimmy Lin (working with Michael Schatz), University of Maryland, presents three design patterns that address designing scalable graph algorithms, and can be used to accelerate a large class of graph algorithms based on message passing, exemplified by PageRank. Experiments show that the application of these design patterns reduces the running time of PageRank on a web graph with 1.4 billion edges by 69%.

Media Production by BAYCAT, a non-profit community media producer that educates and employs underserved youth and adults in the digital media arts.

Today were making some important announcements on the transition of our Search back-end infrastructure to Microsoft, and how this transition impacts the Search APIs and web services we offer on the Yahoo! Developer Network. We are also sharing specific news about several of our other developer services.

Over recent years, Yahoo! has made a commitment to developers by opening products, services, and canvases for third-party innovation. This commitment remains unwavering. For example, we recently announced new canvases and APIs as part of our Zynga deal. At the same time, we have to align our developer offerings with our products and strategy.

Yahoo! Search BOSS

Search remains critical to Yahoo! and were happy to announce that we will continue to offer the BOSS program (Build your Own Search Service). In the not too distant future, BOSS will provide web and image search results from Microsoft along with other search-related services and content from Yahoo!, such as news. In the next

Worldwide spam volumes this year are forecast to rise by 30% to 40% compared with 2009. Spam recently reached a record 92% of total email. Spammers have turned their attention to social media sites as well. In 2008, there were few Facebook phishing messages; Facebook is now the second most phished organization online. Even though Twitter has managed to recently bring its spam rate down to as low as 1%, the absolute volume of spam is still massive given its tens of millions of users. Dealing with spam introduces a number of Big Data challenges. The sheer size and scale of the data is enormous. In addition, spam in social media involves the need to understand very complex patterns of behavior as well as to identify new types of spam. This presentation discusses how data analytics built on Hadoop can help businesses keep spam from