I had the privilege of taking part in putting together a leg of a POC that included the task of parsing a Json data set representing a group of users and doing two things with them. The first, adding them as Sitecore items in what could represent a profile page. The second, creating Sitecore user accounts based on that same json formatted data set.

Fellow Velirian, Nick Dorrough is presenting a series of posts on this POC. Be sure to check out his current post and the soon to be upcoming posts.

Along with the current DXF documentation being a resource, Nick's posts will dive into the details of the POC and working with DXF. So I certainly won’t repeat that information here. What I will focus on is the ability to add functionality of parsing Json and working with Sitecore not only in content creation, but to work in other Sitecore aspects, such as adding Sitecore User accounts.

In initial investigations, it was really evident that my work was not going to be done for me. No Json provider was noticed nor within the Sitecore provider a way of creating users. Curses!

Actually – I kind of thought that might be the case – so into the sandbox to play…..

First I looked into the File System Reader pipeline step provided in the documentation example (http://integrationsdn.sitecore.net/DataExchangeFramework/v1.1/implementing-a-provider/implement-pipeline-step/implement-pipeline-step-processor.html). The process works on the concept of file location and parsing by delimiter. I knew getting the json parser to work was likely more important, so opted to use a local json file. I could always change the way the json was delivered after the proof of concept and likely would need to, based on using the patterns for future solution implementations. I also needed to generate some data representing user accounts. I pulled user data from our GitHub company account and noticed the results needed a little data scrubbing, so at that point there was no turning back from using a local file.

So here is the result of the code for the new converter and pipeline step.

The converter I did was in fact no different than the one in the File System reader example. Except for the template Guid.

For the processor, I noticed the file system example loading “record” elements in a pipeline plugin that contained on object of IterableDataSettings. I knew that likely downstream I would create more work for myself if I changed this approach so, doings as the locals do, I deserialized the json into a List of POCO elements.

//
//read data from file
IEnumerable<GitMemberJsonModel> lines = LoadJson(settings);
//
//add the data that was read from the file to a plugin
var dataSettings = new IterableDataSettings(lines);

So as you can see it is similar to the example. The major difference between the two is obviously the Read Json User File pipeline step.

So when the pipeline batch is executed – the json file is read, json “parsed”, field values are mapped, and like magic we have our Sitecore items.

1st goal of POC completed. No sweat. Also no rest for the wicked, I have another goal to accomplish…

So now instead of writing to Sitecore items I need to create users through the membership provider. So simple…well it took a little bit to find the right fit. I had a few options, some kinda gross and one not gross.

I could have just extended the UpdateSitecoreItemStepProcessor to add Membership.CreateUser after the item update. This is gross because I’ve just couple item create and user creation, which goes against the pipeline concept. Not happening.

I could add a pipeline step within the User Info from File to User Item Pipeline, but then again my entire pipeline has both item creation and user account creation and when calling the pipeline I have to content with having both.

I honestly did do this for a bit for debugging purposes to make it easier to figure out where my data iteration lived and what it looked like for the plugin. But it wasn’t staying that way because sometimes I don’t want a Reeses Peanut Butter cup, sometime I just want the peanut butter.

The converter now stores the Sitecore pipeline step item’s field values as a plugin for retrieval in the processor. Knowing the json record were typical fields in the itemModel – I just grabbed them directly for storage.

Now the processor I based off of the decompilation of the UpdateSitecroeItemStepProcessor. I removed pieces around endpoint retrieval and modified the fixing of the Sitecore item to actually do this:

So last week I had a blast meeting some new folks, seeing some past friends, and deepening relationships with many of you. I had a ton of great stomach bursting laughs and learned a lot from all of you.

One of the many bits of information from last week's Sitecore MVP Summit and Symposium was we get to look forward to this each and every year. SUGCON NA has essentially been replaced with an annual Symposium.

Part of my learning about all of you came with hearing your personal stories, about your travel, about your families, all sorts of stuff.

I anticipated some conversations around user groups, and had some conversations of ideas around an online development conference. Going through those conversations many of your stories stuck with me about family, kids with their tablets and YouTube favorite series, and some of your kids programming. Along with our conversations about Sitecore development, these stories really stuck with me on the way home.

Well, I touched base with Akshay Sura after getting back home. One, to make sure he gave the camera phone a rest, but the other more important reason was to discuss a new Sitecore developer conference idea. Consider the results of the conversation to be what you will read below. Consider this a sort of exploratory effort to gauge interest and ideas.

Knowing that most companies will not send you folks to two Sitecore conferences, we wanted to mix the idea of mixing business and pleasure so we are looking at making this conference a family friendly event. Think of it as a vacation with the family, but getting your Sitecore knowledge gain as well.

We want to have the typical presentation tracks that we are accustomed to seeing at other conferences.

We would like this conference to have sessions geared at teaching aspects of technology (Sitecore related always a plus) to boys and girls of all ages. Gives us all an opportunity to help mentor and teach those coming after us.

We want to have attendee run discussions that are open and collaborative. In some past conferences these have been referred to birds of a feather conversation or "unconference" sessions (thanks Derek Dysart!).

We want to have attendee run hands on labs in smaller group settings.

After session activities that are family focused (game night, etc.) will be required.

We want the conference in a location near a major airport, at a family friendly resort/hotel.

There are other Microsoft family focused conferences which is where the final idea settled in for me. But we don't necessarily want to replicate those. After all we are Sitecore community members, we tend not to follow the typical norms.

We do want more feedback on the idea. Once we can gauge what you as the community are interested in, then we can drill down into the information, start getting people involved in the effort, and officially begin the organization of this conference.

You can reply to this blog, or a much better way to leave feedback is to send myself(@mservais) or Akshay(@akshaysura13) a message or tweet on Twitter. We really want to know if you would consider attending such an event and what else you might like to see.

So I got a gentle reminder from a Sitecore project recently that Solr is going to only return 500 maximum results by default per the ContentSearch.SearchMaxResults setting.

As you can guess, I needed more than 500 results to be retrieved from the index.

So it was as easy as modifying the number and I was in a happier place.

But being me, always looking at the periphery, what if I wanted all results for everything. In older versions of Sitecore you were able to place a '0' for a value and receive the whole collection from the index for your search.

Well that was in older versions of Sitecore, so that didn't go. It actually gives me 0 results. Really useful by the way having a search forced to return zero results.

Regardless through a brief Slack chat conversation Mark Cassidy that the new unlimited results value is actually leaving it as an empty string (""). That did seem to bring everything back so great - curiosity extinguished - right?

Yes, but that flame was sparked again with a comment from Richard Seal. "... just a caveat when doing that. That will send int.MaxValue to Solr as the max rows." For those that don't have that memorized - that value would be 2,147,483,647 for an int32 and for an int64 it is 9,223,372,036,854,775,807. Yes I had to look those up a well.

In our case - int.MaxValue is equivilent to an int32 value. The int64 number was impressive to me so I through it in to both educate and confuse.

Disclaimer: I have not used a decompilation tool to determine the exact interfaces from Sitecore to Solr for this blog, so I do not know how Sitecore addresses things like modification of FETCH-SIZE and etc. with Solr. Those tasks I do when I need to know and for this - I'm on a don't need to know basis.

So I return up to 2 billion records back. That can be a bit taxing.

In my original scenario of increasing the number of SearchMaxResults, I increased it to 15000. Why? Because the feed I was generating from the index had 3,987 (or something like that) pieces of content. Well over the 500.

But then why 15000? Simply for growth of content. New pages, articles, and events will both accumulate and drop out of content. It will take some time for them to hit that ceiling, and when they do - we can change the number.

Going after 2 billion results would still have only surfaced the 3987 pieces of content but it seems a little overhead might be coming for those 3987 records doing it this way.

Being Solr and Lucene are roomates of sort (Solr being built on top of Lucene), it is a bit interesting. As the amount of data increases in the index, I can assume that the algorithm will need an increasing amount of resources to parse records. This is a pretty safe assumption I think. If the algorithm to return resulting sets uses the same algorithm for numeric transformations that the steps should increase as well.

Quote:Because Apache Lucene is a full-text
search engine and not a conventional database, it cannot handle numerical ranges
(e.g., field value is inside user defined bounds, even dates are numerical values).
We have developed an extension to Apache Lucene that stores
the numerical values in a special string-encoded format with variable precision
(all numerical values like doubles, longs, floats, and ints are converted to
lexicographic sortable string representations and stored with different precisions
(for a more detailed description of how the values are stored,
see NumericUtils). A range is then divided recursively into multiple intervals for searching:
The center of the range is searched only with the lowest possible precision in the trie,
while the boundaries are matched more exactly. This reduces the number of terms dramatically.

For the variant that stores long values in 8 different precisions (each reduced by 8 bits) that
uses a lowest precision of 1 byte, the index contains only a maximum of 256 distinct values in the
lowest precision. Overall, a range could consist of a theoretical maximum of
7*255*2 + 255 = 3825 distinct terms (when there is a term for every distinct value of an
8-byte-number in the index and the range covers almost all of them; a maximum of 255 distinct values is used
because it would always be possible to reduce the full 256 values to one term with degraded precision).
In practice, we have seen up to 300 terms in most cases (index with 500,000 metadata records
and a uniform value distribution).

Lots of words isn't it? I think it is just easier (at least for my math skills) to generate calculations around content and expected returned results. Breaking up indexes potentially by ontologies and related categorizations of relationships will also assist with limited the scope set being returned.

Best said from an old man I once had the pleasure of having conversation with - "Take only what you really need".

During the past year, I found some different ways to preach to the goodness of Sitecore through involvement in the Sitecore! Experienced work, posting through a few different blogs, and conversing with Sitecore customers.

I did more with SPEAK than I imagined through working on XCentium's upcoming Vault product, I learned from each of our guests on the podcast, and really focused my personal interests into career development of myself and a few others.

This year the Sitecore site took hold and the Sitecore Community Slack channel was born. Watching both of these flourish this year has been really fun to watch.

Now there are more MVPs than ever before - 221 of us now. Going through the list some regular faces and a lot of newer folks. Group pictures are really going to suck getting everyone in the shot. A drone might be needed this year to take that group photo.

The community is getting better and better.

The weird for me, besides the weird zombie drunk guy in New Orleans trying to spit food on Mark Stiles, Dan Solovay, and myself, was the amount I took on and the speed of burnout that was incorporated. This year was the first time I felt I needed a hiatus from career focused and some personal endeavors. That was a bit tough for me to do, but in hindsight was necessary.

Going forward, changes are a coming for me this year. New directions around career and life are going to be interesting for me.

Sitecore! Experienced ideas are flowing and getting on the board to execute and finish...

Module, big data and whitepaper ideas are making me salivate like a dog hearing a bell..

We as co-creators are sending off the first of our offspring to university - our work to laying adulthood groundwork is complete and I get to enjoy the role of adviser instead of composer

Embracing and exercising my non-logical thinking more which produced a by product of strengthening my logical thinking. A little odd kinda like a politician that thinks for everyone else before themselves. (BTW - that politician thing doesn't exist in the United States. I don't want people to have false hope about money and politics.)

So here is to Sitecore's 2016 season that will bring the future MVPs of 2017...let's tear into this with a little more swagger in our step, shall we?

I just wrapped up a presentation (admittedly clearly not my best) for the Milwaukee Sitecore User Group where I gave an very brief overview of SPEAK and I did a little Betty Crocker baking of an existing Sitecore Marketplace module from Sheer UI to SPEAK.

The prezi is here. The MKE SUG presentation video is here. Finally the summary video I did as a backup is here.

A small Sitecore! Experienced video is here - our first Tangible Knowledge series piece. These are open forum so if you got something uniquely Sitecore to share see us on the Sitecore! Experienced site.

SPEAK has some portions that give it some heat. Let's review some of these nuggets:

Config files:

Sitecore.Speak.config -> Contains a series of SPEAK settings around requireJS files, caching, script minificaton, etc. Play with this file at your own risk. Big takeaway in this file is the addition for pre compilation of SPEAK views and the custom SPEAK handler which is critical for component rendering.

001.Sitecore.Speak.Important.config - Important enough to be at the head of the list. All the client piplelines are defined here for the client processing -clicks, layout rendering, scripts and stylesheets, and bindings.

Sitecore.Speak.ItemWebApi.config -> Pipelines. The word you're going to here for a bit more. So this ties in the pipelines related to item search for SPEAK. Item requests, searchs, and property retrieval are defined here.

Sitecore.Speak.Applications.config -> Some settings and pipelines that deal with dialog presentation overrides , the media browser dialog (XML control override), and logout functionality around SPEAK.

Sitecore.Speak.Components.config -> Some of those things, you know - pipelines. Rules, controls, styling are defined here.

Sitecore.Speak.LaunchPad.config -> No pipelines! Instead a processor to work with SPEAK log in functions and a setting to enable personalized frames with Analytics.

Sitecore.Speak.AntiCsrf,SheerUI.config - Looks to be a rules filtering limitation - but don't quote me on that one.

Sitecore.ExperienceEditor.Speak.Requests.config & Sitecore.ExperienceExplorer.Speak.Requests.config - processes related to content author activities to verify that actions on the client can take place, retrieve specific information and perform specific processing actions. Looks like a lot of tool setup.

So knowing where things are defined for life in Sitecore will come in handy when having issues with very customized and complex controls.

In keeping with the theme of pipelines, we just don't have them for the code-behind. In your Sitecore instance file structure, go to website/sitecore/shell/client/Speak/Assets/lib/core/<pick your version here, we are looking at 2.0>. The sitecore.js file begins the journey of functions that are the girders and bolts for the SPEAK framework. You can read this at your leisure.

Ok you really don't need to spend leisure time reading this, read Moby Dick, White Fang, Grapes of Wrath, or a great piece of literature.

Do however take a little bit of non-leisure time to review this script as it will give you some insights on the client side composition of SPEAK and where frameworks like jQuery, Knockout.js, and Backbone.js come into play and also where their wings may have been clipped/enhanced for SPEAK.

You likely will never want to make changes here, but always good to know what is going on where.

This URL -> https://doc.sitecore.net/speak = your friend when working with SPEAK. You are going to make a lot of other friends too. Several Sitecore employees, Sitecore MVPs, and dedicated developers have written blogs on some trials and successes with their speak efforts.

The most difficult part about working with SPEAK has been knowing what each of the components can and can't do. Even with the improvements in documentation, you are going to have needs and wants to push the provided components further than they were designed to go. You may even find the need to have to create something brand new (I encourage you to do so frequently).

There are a few good tutorials on how to get started with SPEAK and I will not reiterate that information here. I will though share with you some honest advice around my using SPEAK for XCentium's Vault product and other little tidbits I have worked on.

1. Don't screw with the main layouts. Yes that is a technical term that I made tamer than the original word. Sitecore has put these in place for a consistent feel for all customers. I haven't seen too many hard and fast UI best practices for consistency from the controls provided, but the dashboard pages and dialogs have a definitive sandbox to do you thing in. Play in the sandbox as much as possible.

2. The learning curve is certainly inclined. Even if you have a solid understanding of the stack behinds SPEAK (I consider myself mediocre here), there are things that you can, can't, should, and shouldn't do. Trial and error around building stuff is the best teacher here.

3. The premise of SPEAK is built on concepts you should know as a Sitecore developer. Placeholders, templates, renderings, etc. It's just a different composition toolbox and space.

4. Sitecore Rocks is another friend of yours. In fact you really can't do SPEAK without it. I did though have to go into the Content Editor within the Core database to verify some checkboxes. They didn't set very clean for me in Rocks.

5. Estimation is futile at first. That incline I mentioned before, it humbles you. Starting out, especially if building custom SPEAK components, you are going to mis-judge the time it will take you to accomplish tasks. If you are a consultant, and promising custom SPEAK based functionality, you may want to decide on some what-if overage scenarios. At minimum, give yourself plenty of runway to complete what needs to be done. If possible start with smaller tasks to begin to get a structural feel around the environment.

6. Understand the concepts behind backbone and knockout. While you will not want to rely on them entirely in case the stack changes later, you most likely will want to leverage the architecture the best you are able to keep your components and overall flow clean.

7. Listeners are king.

8. Rules help with component to component interaction. If that fails, utilize your page code capabilities to facilitate that communication.

9. Keep your code generalizations in nuggets for re-use but expect that each "page" entity in SPEAK will require its own unique capabilities.

10. Provide as much feedback as possible. Sitecore has invested time and energy into this as a future piece of the puzzle. The more voices they here about issues, improvement ideas, etc., the better it is for the entire community. The real good part - they are listening.

11. Your browser development tools are another really close friend you will make. Learn them well.

I hope you found some wisdom in my words. Good luck with your adventures and share what you are doing.

Here are some of the best places for SPEAK information as of now:

Sitecore Support, the Sitecore community (community.sitecore.net, the Sitecore slack channel (#sitecore-speak), Twitter, etc.), your local Sitecore User Groups - talk to other Sitecore folks to see what they are doing.

It's done. We did something that we weren't sure had much merit, how much work it was really going to be, nor did we think we originally were going to do more than 3 episodes.

Jamie and I took on these nine episodes and had a lot of fun doing it. Sure there were many nights staying up till 2-3 AM to shoot interviews, marathon video editing sessions, and countless challenges along the way, but it was something that two people in the Sitecore community were able to put together and move forward.

This was very garage punk band stuff for me. The sessions were pretty raw from a humor standpoint. Most material cut from the final versions of their episodes will never see the light of day. The guests were amazing. For many guests it felt like we knew then from childhood making the process so much more enriching than I could have ever imagined.

But like everything in life, you can only ride the wave for as long as it lasts, and we felt like this wave crested and came to shore. It is time for both of us to determine what is next for Sitecore! Experienced. We know this much, it won't be a podcast.

We wanted to go on to other projects and endeavors within and outside of the Sitecore community.

Jamie and myself had developed this friendship from merely working together within the same team. We took that friendship and this collaboration to allow others to share what they do in the Sitecore community, while we selfishly got to hang out, tell bad jokes, catch up on family news, and have some fun and laughs along the way.

While we will still be doing some things together in the future (and we really don't know what that will be yet) I would like to encourage the community, which we think is one of the better software development communities out there, to continue to extend, share, and find ways to share by working together.

The benefits of doing so have now been repeated a few times in the community since our episode #1 alone.

So thank you for all your support and your viewership.

You can find me soon on the next wave! I don't sit still for very long.

I love talking to young folks still in school about tech. In many ways they are so much smarter than I was, or in some cases smarter than I am currently. The sad part is there is not as many of them as there used to be.

The single one question I get the most, other than how do you become a Sitecore MVP, is "my company does not have Sitecore but I want to learn Sitecore. How can I get a copy to practice on?".

Yeah - not so much an easy task there due to licensing. I think without working for a company with the right licensing or working or a Sitecore implementation partner you are going to have a difficult time getting your hands on a copy of Sitecore.

However, if you really want to work with Sitecore and are dedicated to do so - I would seek out those opportunities. If you are unable to do so, like if you are a student not yet seeking employment, I usually provide the following advice:

Understand both the Microsoft ASP.NET webforms and MVC technical stacks. Each framework has it’s own unique composition model to understand. Since Sitecore sits on top of the .NET framework, any knowledge learned here will apply.

Get some working knowledge of content management systems. You want to understand content topologies, dependencies, and general authoring process practices.

Understand how the web works. This really means how do servers communicate, how does HTTP really work, sessions, cookies, and overall state, and it never hurts to understand TCP/IP and how to break apart packets if necessary.

With Sitecore - understand the marketer. Take some general marketing classes and understand their terminology. Understand events, campaigns, and how they use information gathered.

Understand HTML/CSS/Javascript and related frameworks for front end presentation. You don’t have to be a front end wizard in a lot of cases - but you need to have a working knowledge of the skillset.

Read blogs and watch videos. The Sitecore community is by far the best tech/product community I have ever had the privilege to be associated with. There are some of the smartest folks I have had the pleasure to associate with at your disposal for questions and help. The great part about all these smart folks - they love to help others to do what they do - completely unselfish. Twitter and Stack Overflow are mentoring playgrounds for you to become involved in.

Attend one of the many and constantly growing Sitecore user groups. If you don’t have a local one in the area - Hedgehog sponsors the Sitecore Virtual User Group and many groups like the Milwaukee Sitecore Users Group stream their meeting presenters.

Finally if you can afford it, there are some great offerings for training from Sitecore. These can be local or online based training.

In my opinion, it would be great for universities to establish courses around CMS, particularly Sitecore, as part of an advanced development curriculum. If that happens I would love to teach that class for sure.

DISCLAIMER: I work at RBA. Everything here, however, is my personal opinion and is not read or approved before it is posted. Opinions, conclusions and other information expressed here do not necessarily reflect the views of RBA.