clientdev: new XML ticker output (idea)

This is a little complicated, if you spot any mistakes *please* /msg me.

Introduction

Many people have written little scripts and utilities designed to work on information from e2. At first, people extracted the information they required from the HTML you and me view every day on the site. However, this was buggy and required modifying for every theme, upon anyone changing the layout, etc. So along came the old XML tickers, still retrieved via HTTP, which you can find at the top of Everything Data Pages. And so did they provide the basic information we all love, and scripts and utilities such as the E2 Node Tracker and the Java Chatterbox used them. And also provided was the XML displaytype, which edev people could use to view their own nodes. And the magnificent nate made this buggy displaytype available to all to view e2nodes and others writeups, but it was buggy and exported minimal information. And so wasn't incredibly useful.

And so, the beloved JayBonci, after much prodding from me and my quest to write e2client, has developed a new set of XML tickers! They produce well-formed, valid XML, and replace the old tickers entirely, although they're still there for backwards compatibility. But you need not use them any longer! They have been cast off a cliff, impaled on rocks and left there to die, as they deserve! The new tickers export a lot more information, and have been designed so that, when correctly used, they add very little to server load. And displaytype=xml has also been replaced! We now have displaytype=xmltrue, which exports all available node information from a variety of nodetypes.

One requirement of all clients is to assume a server error on malformed elements. This is because of in case of an actual server error (yes, we all know how perfect our code runs and how infrequently one of the boxes does not run out of memory), the output will in fact be mangled. If you encounter a server error, please try to reload the page in a minor amount of time, and fire a decent user error off. This is a general limitation to ecore in the sense that this is only xml-ish behavior, and does not reflect any underlying changes to the core.

Note that almost everything - node titles, document text, etc - is passed through the encodeHTML function before being output. This replaces < with &lt;, > with &gt;, & with &amp; and " with &quot;, which is obviously necessary for it to be valid XML (as, for instance, the HTML of very few writeups would turn out to validate as XML). So you'll probably want to reverse this before actually using the data.

Also note that you'll probably want to pass a login cookie along when you're requesting all this stuff! You can either construct it manually (it's a pretty simple structure, the password is just crypt()ed, with the first two characters of the username as the salt), extract it from an op=login, or extract it from a web browser's cookie.txt. :)

This writeup fully documents this new e2 XML functionality. I will be updating it as necessary as we update and add to the exports and/or tickers. If you'd like to discuss all this XML stuff, we have a usergroup - clientdev - for just that purpose! /msgN-Wing or JayBonci to be added. Alternatively, feel free to /msg me questions.

Node Exports

If you're in edev, you'll probably want to look at the code for this, which is in node xmltrue page. Called from this is the code for the header (xmlheader), the footer (xmlfooter) and the code to render the actual node (formxml, which just calls whichever one is appropriate for the node in question, see the links in the list below).

At present, we have exports for the following types of nodes (if you're in edev, you can view the code):

To retrieve the XML, simply add a displaytype parameter with a value of 'xmltrue' to the URL of your request. This would typically mean adding '&displaytype=xmltrue' onto the end.

So, for instance, if you were trying to retrieve the node Butterfinger McFlurry your URL might be http://www.everything2.com/?node=Butterfinger+McFlurry&displaytype=xmltrue. If you already had the node_id (which is 980113) and wanted to avoid the need for the server to look the name up, you might use http://www.everything2.com/?node_id=980113&displaytype=xmltrue.

But wait! If you're like me, you don't want the server to parse the links for you, you want them left in [link] form, with the square brackets. In that case, the 'links_noparse' parameter, which must have a value of '1' to work, comes to your rescue! So you'd request http://www.everything2.com/?node_id=980113&displaytype=xmltrue&links_noparse=1.

Okay, so, we've worked out where the code is and how to request the XML. Now you'll be wanting to know the format of the xml... well, here's an example.

Okay. The first line is just a generic XML header, of course. You can ignore most of the parameters of the node tag for practical purposes; you'll probably just want the node_id and the createtime, both of which should be self-explanatory. The contents of the node tag are of course what is important: the type is the nodetype, the title is, well, the title and the author tag provides both the node_id and the name of the node creator.

The node-specific data is discussed in seperate sections, below.

Node Exports: e2node/writeup

The node exports for e2nodes and writeups are very similar, the main difference simply being that only one writeup is exported for a writeup node, whereas many are exported for an e2node.

Both e2nodes and writeups have 'softlinks' and 'firmlinks' tags. These contain of a set of e2link tags, which consist of a node_id parameter and the node name as the contents of the tag; for example: <e2link node_id="1353299">VDMSound</e2link>. The softlinks are exported in order, and the same number are exported as are available; so if you're not logged in you will recieve less.

e2nodes have a 'nodelock' tag, which, if non-empty, contains either the softlock text or text which states that the account being used is locked so it cannot create writeups. They also have a 'sametitles' tag, which can contain 'nodesuggest' tags. These tags have a 'type' parameter and contain an e2link tag pointing to the target node, in the same format as above. 'nodesuggest' tags representing users may also have a 'useralias' tag which means that the user is a chatterbox forward for another one - the aliased user is detailed in an e2link tag inside the 'useralias' one. An example of this is shown below.

An individual writeup will just contain one writeup tag, whereas an e2node may contain zero, one or more writeup tags. The parameter 'no_doctext', if provided and set to 1, means that the contents of the 'doctext' tag will not be provided. An example of a writeup tag is shown below:

The 'marked' parameter is normally 0, but is 1 if the writeup is in the node row weblog, normally known as being "marked for destruction". The 'parent' tag points to the parent e2node, and is only really useful in writeups. The 'reputation' tag only appears if you've voted on, or are the author of, the writeup in question. I'd hope the rest should be pretty obvious from the example.

Node Exports: user

This is pretty simple. I'm just going to give you an example and let you work it out from there. The parameter 'no_doctext', if provided and set to one, means that the contents of the 'doctext' tag are not exported, to save bandwidth if you don't need it. The 'image' tag is empty if the user has no homenode image. The 'cools' tag doesn't appear if the user hasn't cooled any writeups. Note that the creation time is exported in the node header, and hence is not duplicated here.

Note that groupmembership only exports edev, gods and content editors, just as it is on homenodes. Why? Because those three groups are already loaded when we get to this point, as they're used by various other bits of the code on each page, which means no extra server load.

Node Exports: superdoc

Okay, this is icky. When a superdoc is requested, the code is run, and the results placed in a 'superdoctext' tag. You should be warned, however, that many superdocs manually generate URLs, and as such, links_noparse may not work.

There are two possible parameters which can be used here - if you want to use either of them, it's recommended that you pass them with every request, because you obviously usually don't have a way to work out whether a node is a superdoc or not before you request it. The parameters, both of which have to be set to '1' in order to work, are 'no_findings' and 'no_superdocs'. They both do the same thing, the first just for the Findings page and the second for all superdocs, which is to disable generation and output of the superdoctext tag. This reduces server load for those cases in which you don't want the pages - clients will probably just use no_findings, as they'll be using the Search Interface (below) for this purpose (for instance, when e2client retrieves the Findings superdoc, it automatically sends another request to the Search Interface, so it would be pointless to generate/download the superdoc text).

Node Exports: usergroup

Again, this one is just an example. The contents of the 'description' tag are not exported if the parameter 'no_descrip' (which must, as usual, be '1') is provided. The 'weblog' is used for such things as edevify! and clientify!, and is only exported if the logged-in user is a member of that usergroup.

Node Exports: room

If you hit a room node, you meet the entry criteria (eg, only gods are allowed to enter Valhalla, only M-Noders in The M-Noder Washroom, etc) and you're not Guest User, then you're moved into that room. The returned xml has a 'canenter' tag which is 0 if you didn't enter the room for some reason, or 1 if you did (the usual case if you're not a guest).

Tickers

Tickers are special pages which provide XML information, mostly data required for rendering nodelets, but also other information such as user search and clientdev client data. They all being with a standard XML header tag (plus maybe an inline DTD or schema or whatever), and then are all pretty much unique.

You should request them based on their node_id. And you shouldn't hardcode these node_id's into your code, if at all possible, but should instead use the Interfaces Ticker which is documented below.

Note that you can't request these with the xmltrue displaytype, you'll just get the xml header for the nodetype 'ticker'. They must be requested with the default ('display') displaytype, which you can do by simply not including a displaytype parameter with your request.

For each ticker I've provided a paragraph of explanatory text, and then example 'real' ticker output. Note that in most cases I've stripped the output down to just two or three items where there are many, and that I've obviously modified some of them considerably. :)

Interfaces Ticker

This ticker is also available via '/interfaces.xml', which is an apache alias for the node. Why? Because this ticker is a site-independent way to retrieve tickers. You simply retrieve the xml file, lookup the node_id of the ticker you want in it, and then request that node_id. You obviously should just request this once per session, or even just request it once and then cache it.

Since the server move, interfaces.xml has not worked, presumably due to the removal of the apache alias.

Each ticker documented here has the interface name, as provided by the 'iface' tag of each 'xmlexport' tag here, provided. So in order to download a ticker, reference your interfaces list, find the required one, get the node_id, and then download it.

Client Version Ticker

See clientdev: Registering a client for information about client registration. This ticker simply exports the information about all registered clients, mainly so that clients can automatically detect when a new version is available and suggest upgrading to the user.

Other Users Ticker

A list of users logged onto e2. An optional parameter 'in_room', with a parameter of the node_id of the room in question, restricts the output to users in a certain room. A parameter 'nosort', if set to 1, means that the output is not sorted. Please enable this and sort client-side by the 'xp' parameter, in order to reduce server load.

Universal Message Ticker

Okay, this can be used to retrieve two types of messages: chatterbox messages, and your message inbox. It takes a bunch of parameters:

'msglimit', an integer parameter. This should contain the id last message you have stored locally; it will only return messages which are newer than this one. If not present, all messages are returned.

'for_node', an integer/text parameter. This should contain the node_id of the room and/or user you wish to retrieve messages for. If you set it to 'me', it will return your own messages. It will not work for retrieving messages from any user except yourself, nor any room except the one you are in. If not present, messages from the 'outside' room in the chatterbox will be returned, which will NOT WORK unless you're in that room. If you're not logged in, ie logged in as Guest User, you can retrieve the contents any 'public' room (this is currently outside and political asylum only).

'backtime', an integer parameter. This should contain either '5' or '10', depending on whether you want the last 5 or 10 minutes of chatterbox messages to be returned. This only works if requesting chatterbox messages (ie, from a room).

'nosort', which should be set to '1' if you don't want the results to be sorted. Please enable this and sort messages by their ids client-side, to reduce server load.

'links_noparse', which, as with the xmltrue stuff, stops links from being transformed into <a> tags.

Personal Session Ticker

This ticker updates your session, and displays useful data like your personal nodelet, the votes/cools you have left, the amount of karma (Golden Trinkets) you have, and updates/displays your borg status. IN ORDER TO BE UNBORGED, YOU MUST EITHER REQUEST THIS TICKER OR CAUSE THE EPICENTER TO BE DISPLAYED (typically by downloading a normal (ie, web-based) e2 page). The 'forbiddance' character displays a userlock, which is the text displayed like a softlock if your entire account is stopped from posting writeups. The 'xpinfo' section only appears if your numwriteups or XP has changed since this ticker and/or the Epicenter were last executed. Note that if you're not logged in, only the 'currentuser' and 'servertime' tags are produced.

Random Nodes Ticker

The Random Nodes nodelet, but an XML version! Along with that randomish witty phrase you've grown to love. You get 12 of them each time it's refreshed (every minute or so), and JayBonci says "It gets fed off a nodelet called Random Nodes XML Feed, so feel free to hit this nodelet as often as you like, it's quite cheap, I can assure you. It's also fine to hit from Guest User".

Search Interface

A tool for searching the site using keywords, just like the search textbox at the top of all e2 pages. Pass a parameter 'keywords' with your keywords and a parameter 'typerestrict' with the type of node you want returned (probably the default which is used if you don't provide a 'typerestrict' parameter at all - 'e2node').

Scratch Pad Ticker

Returns a scratchpad, by default yours, however you can provide a 'scratch' parameter, which consists of the node_id of the user whose scratchpad you wish to view. You cannot view it unless the user has set it as shared, and you cannot view it if you're not logged in. You can also pass 'links_noparse', as with the xmltrue interfaces.

Cool Nodes Ticker

Returns a list of cooled writeups, with optional parameters as listed below.

'writtenby', an integer parameter, containing the node_id of the user whose cooled writeups you wish to view. If not provided, displays cooled writeups by all users.

'cooledby', an integer parameter, containing the node_id of the user who cooled the writeups you wish to view. If not provided, displays writeups who were cooled by any user.

'limit', an integer parameter with a default and maximum value of 50, which specifies the number of writeups to list.

'startat', an integer parameter, which contains the location in the list you wish to start the list at, commonly used when paging through a large list (as a list of cooled writeups is likely to be). So, for instance, you'd retrieve the second page with 'startat=50', assuming you retrieved the first with 'limit=50'.

Editor Cools Ticker

Editor Cools, otherwise known as 'Endorsements'. One integer parameter, 'count', which defaults to 10 and has a maximum value of 50, which specifies the number of writeups to return. Sorted by time of endorsement.

Everything's Best Users Ticker

The XML equivalent of the Everything's Best Users superdoc, along with a shiny 'ebu_noadmins' parameter, which, if set to 1, doesn't include gods, and if not set or set to anything else, does include gods.

Node Heaven Ticker

Retrieves the contents of your Node Heaven. An optional parameter, 'visitnode_id', means that the only writeup displayed is the one matching that node_id, if one exists, and that the contents of that writeup are also provided. The writeups are sorted by title unless the 'nosort' parameter is set to '1'. 'links_noparse' is also supported, and works the same as it usually does.

User Search Ticker

The equivalent of User Search, but in XML form! Supports the following truly amazing parameters:

'searchuser', the username to display the writeups of. If the user doesn't exist or this parameter isn't provided, it defaults to the logged-in user.

'startat', for use when paging through the list bit-by-bit: defines where in the list of writeups to start output.

'count', which defines the amount of writeups to output at a time, with a default value of 50.

'nolimit', which, if set to '1', means that 'startat' and 'count' are disregarded and all writeups written by this user are displayed. Obviously, try to avoid using this, it involves quite a lot of work server-side.

'nosort', which, as ever, should be set to '1' to disable sorting and hence reduce server load. DO IT CLIENT-SIDE!

'sort', which has three possible values: 'rep', which sorts by reputation, 'title', which sorts by title, and 'creation', which sorts by creation time. Obviously this doesn't work with nosort.

Time Since Ticker

Returns the time since a set of users were seen. Pass it either the parameter 'user', with a comma-delineated list of usernames (eg 'user=JayBonci,TheBooBooKitty,nate') or the parameter 'user_id', with a comma-delineated list of user_ids (eg 'user=4653, 452753, 296735'). If you don't pass it either of these parameters, it details to returning information about the logged-in user. ascorbic notes that it's also quite useful for matching usernames with user_ids...