The Ingestion Box in the reference architecture is displayed as the smallest box. However, this is the component that integrates with all the available data sources. This tends to be among the most complex and time consuming task, but tends to be relegated to a lower priority which is a big mistake.

One needs to prioritize the data sources that generate maximum value and ensure we can ingest the data into the Big Data platform for subsequent “cool” analytics.

In my experience, it is also extremely important to have a robust User Interface for the ingestion section. Otherwise, there could be a series of manual steps leading to errors and ingestion of “bad” data that will minimize impact of subsequent analytics.

Just released a 30 page site built with bootstrap. Simple & extremely fast to build. built in less than 1 week with 1 developer new to the bootstrap framework. Site works very well on iPAD, Android OS and iPhones.

As I review the successful (defined as on-time and within budget) & unsuccessful projects that I have led over the past 10 years, a common thread has been that successful projects had at least 80% complete prototype prior to start of any development work. This is spread across both Agile and waterfall methodologies.

The prototype allowed the Business Analysts to “touch” the product and better visualize & communicate what they expect of the product.

To get a project kicked off, start with a UI Developer and a Business Analyst. Refine requirements till you are 80% complete and then bring Software Architects.

This may seem contrary to several Agile approaches, but in my perspective it is a modified Agile approach that has worked for my teams. The agility is divided into 2 phases – Agile requirement gathering (with UI developer & business Analyst) and then move to Agile software development.

I have also noticed that bringing software developers into the process prior to the product vision being ready, tends to compromise the product vision and it is best to involve them in architecture & development once the vision is clear.