Aug 13, 2008

RTE in TYPO3 sections: a nasty bug

I thought I should publish a little story how the bug #8232 was solved. It took me 11 hours to make a fix and the story may help other developers to solve similar problems.

But firsts I want to give you a little insight about sections and RTE in TYPO3.

TYPO3 AND SECTIONS

Sections in TYPO3 is a way to have repeating structured elements in forms. Mainly it is used by TemplaVoila and its Flexible Content Elements (FCEs).

For example, think that you have a product and a number of its characteristics. You want characteristics to be defined in the structured way. You want to have a title and a link to the page with details. You can do that all in RTE but then your data and presentation are bound together. If you have many products and want to change presentation for all of them, it becomes a problem.
TemplaVoila separates data from presentation. You define data separately and can assign any template to represent this data. So changing the representation means simply changing the template in one place.

Suppose, you created a section that contains title and link using TemplaVoila:

As you see here we have a section named “Feature list”. This section can have repeating elements named “Feature”. Each element consists from a title and a link. Pretty easy.

When we go to TYPO3 Backend, we first see the following:

Clicking the icon to create a new feature creates this new product feature:

And so on. There can be many features and TemplaVoila will render them according to the template. There can be even different templates if you wish to render the same data differently on different pages.

You can also move or delete elements using icons on the top right of each element.

Technical view: how sections work

Backend forms can be very complex. Therefore creating the whole form in JavaScript would be very hard. Therefore sections “fake” the form for each possible section element. Next that faked form is converted to a template. The template is added to onclick handler for the “Add new” link. When the link is clicked, JavaScript code is executed and inserts the template to the certain position on the page. The code takes care to modify static id in the fake form.

All this works fine until you attempt to use the Rich Text Editor (RTE) in the section.

Problem with RTE in TYPO3 sections

While most elements are native to the browser, RTE is not. It requires a lot of extra work for initialization of every RTE instance.

Typically RTE is created from a simple text area. Thus if something fails, user still see the text area with HTML source in it.

RTE is initialized correctly in normal Backend forms because the form always knows how many RTEs will be there on the page. In contrast, sections are dynamic and each RTE instance has to be created dynamically and initialized properly. In addition, each RTE must have a dynamically generated id attribute. The code for RTE should be able to find all those dynamically created RTEs and process various events for them. The most important event is the save event. The other event, now section-specific, is the delete event (when section element is deleted).
JavaScript code for the HTMLarea RTE was not ready to work with dynamic sections. In my personal opinion this is no one's fault but lack of communication. Many developers work independently on TYPO3 and they do not share the ideas and technical details of their code with others. This is also true for RTE and sections. Thus the developer of sections did not know about RTE complexities and RTE developer was unaware of the dangerous changes in sections.

As a result of this lack of communication RTE code could not handle dynamic nature of RTEs in sections. Certain parts of JavaScript failed when an attempt to create RTE was made. Further the code to renew login status was not called due to this JavaScript error (the whole block was aborted due to one error). Internal JavaScript login variables on the page were not set. When user pressed “Save” button, JavaScript code found that login time is zero and treated it like “login expired” case showing the login prompt.

Further, if there were more than one RTE, RTE JavaScript still assumed that there is only one RTE on the page. So every next RTE overwrote the content of the first RTE. As a result after saving the value of the last RTE was placed to the first RTE and all other RTEs were empty.

Deleting also had problems. Firsts, if RTE is dynamically deleted, RTE JavaScript code was unaware of it. When save button was pressed, JavaScript tried to contact that deleted RTE and complained that RTE is missing.

Fixing the problem with RTE and sections

After discovering the source of the problems, it was necessary to define how they could be fixed. Simply going to fix them would not do any good because the nature of the problem is complex. Backend forms, dynamic sections and lots of JavaScript in RTE only added complexity to the problem.

Firsts, I decided to fix the problem with deletion of items. For that I had to add extra code to RTE to check if RTE was deleted. A new object variable was created that holds a boolean value to tell the code if RTE is still alive. But this alone could not solve the problem because I still had to identify what editor to mark as deleted.

Next I went to replace id for each RTE dynamically at runtime. This took most of my time. RTE used the counter from TCEforms to address each RTE. When a new RTE was inserted to a normal form, the counter was incremented to address the next editor. Obviously for sections the number was always 1.

So I changed the code to use “length” attribute of the JavaScript array with RTEs. That immediately allowed me to create RTEs with a proper internal number. But I still could not identify the RTE when it was removed. I had to make a connection between RTE and its containing block.

I added the id of the text area to the RTE. It looked like this would work but it did not. Next I remembered that TCEform replaced that id dynamically. Doing that for RTE looked like a solution, so I did that. It may sound simple but it took extra several hours. RTE JavaScript code is long, it contains a lotf characters that would break the onclick handler. So I had to escape that. Finally deletion worked.

Next the same part of work had to be done for saving. This was more difficult. I had to spend another several hours in the debugger looking to the code similar to this one:

That code was executed in “eval”, so true debugging was not possible. I could only make a step and examine variables in the debugger to see what went wrong this time. And restart the debugger. And so on. That took ages.

The worst problem was that it was too easy to screw up something. I had to make diff again and again. Code just worked, small change and nothing works. The change looked logical but it did not work. So I had either to change it more or to revert to the last working solution. But in the end it all worked!

I did it.

Concusion

While I am proud and happy that I could solve it, I am still sure that such things can be avoided. I used to work in several large companies that develop commercial products. They have one advantage: they truly work in the team. No member of the team will go his implementation in the way that will seriously break the work of others. Every part of work is documented and reviewed. Technical designs are created before the coding actually starts. Other team members review those designs and spot potential problems early. TYPO3 and many other open source projects lack it. It is generally not fun to do designs, etc. Implementation is much more fun. However I believe that at some point in time will will come to the proper planning.

When I wrote a book about TYPO3 extension programming, I dedicated the whole chapter to extension development planning. It pays back to plan your work. This is my main point and moral from the story about bug #8232.