Monday, October 31, 2011

Please Excuse the Mess

It’s update time. The season is over and a slower news cycle is upon us. It’s time for me to roll up my sleeves and work on the site. First up is the transition of our Content Management System (CMS) from Expression Engine‘s 1.x series to its 2.x series. I had hoped to accomplish this upgrade last March but, unfortunately, some of the third-party scripts the site uses had not yet been updated to the new version, my personal life was much too busy and, frankly, I did not have sufficient funds to pay for a server upgrade and all the script upgrades. Although money is still tight, things aren’t as busy and all the scripts I need are now available. It’s time to get cracking.

The CMS software was almost completely rewritten in this update. As a result, many of their native and third party scripts, and quite a few of my own scripts, either need to be updated or replaced with something new. I’ve already rewritten my homebrewed scripts. Unfortunately a few third-party scripts have no direct upgrade path, so I will need to pay for a replacement or I will need to write a replacement myself. Luckily many of the replacements are actually superior, so in a lot of cases we’ll end up with better functionality as a result.

Getting BBTF updated is a rather large, time-consuming undertaking. I’ve done some trial runs on my development server and I must say the upgrades are not trivial and the process is often problematic. I’ve learned that the upgrade is better done in steps rather than trying to do it all at once by stringing together a series of upgrade scripts. It is also far easier to troubleshoot problems when the process is done piecemeal. This does cause a few problems because, with my schedule, some portions of the updates will need to be done over a number of days. Although I will try to make the transition as inconspicuous to you as I can, my tests suggest some things may not be available (like bookmarking, categories) for a longer time than I’d prefer.

As in most anything, planning makes perfect. In preparation for the conversion to 2.x, a few items (polling, for one example) will need to be disabled. To keep you apprised of problems, I am creating this thread specially for the upgrade process. Although most of the upgrades will be done in the wee hours, there may be times I will need to take the site offline for short periods during the day. I will try to post a notice at least 24 hours in advance of any planned outage.

With the size of the site’s database I have found less problems crop up during the upgrade when I reduce the number of fields in a few of the tables. Since changing the structures of some of the underlying databases also will make the upgrade of the archives and search features easier, I have decided to make most of these changes prior to the conversion. Again, from my testing, this changeover will, for the most part, be transparent. Nevertheless, I warn you that a few of the older/lesser trafficked blogs may go offline for short periods. I also warn you that the site will most likely be offline for most of the day I run the vendor’s conversion script. I will try to schedule this downtime in late November well before the Winter Meetings. Once again, I will try to give you sufficient notice to lessen the shock (especially for our most hardcore Primates).

Once the conversion to 2.x is completed I will incrementally update the site. Unlike past upgrades I will try to keep the site’s architecture about the same and will simply incorporate new features into the existing site design. My plan also includes upgrading the site’s servers after the holidays. As I mentioned above, I had hoped to upgrade the servers last March but unfortunately with a wedding on horizon, I didn’t have the time or money to get it done. In the interim I was able to tweak the server configurations to get the site’s performance up to a passable level. I fear, however, with some of the features I have planned, our current servers will not be up to the task. I will do my best to get our servers upgraded this January.

Anyway, with less stuff going on in my personal life this winter, I hope to use my time wisely to make the site more useful for next season. I hope you will find my additions worthwhile. Thanks for your patience through the process.

Jim

UPDATE: 12/9/2012 - The updates are underway. Due to requests from the Hall of Merit regulars, some of the updates were postponed until after balloting

Reader Comments and Retorts

Statements posted here are those of our readers and do not represent the BaseballThinkFactory. Names are provided by the poster and are not verified. We ask that posters follow our submission policy. Please report any inappropriate comments.

Apologies if this has already been addressed, but is there a reason why external links are now being redirected via the "http://www.baseballthinkfactory.org/?URL=" prefix? It's now very difficult to make out what a link is by hovering over the linked text, and it also breaks the browser's color scheme for new and visited URLs.

Apologies if this has already been addressed, but is there a reason why external links are now being redirected via the "http://www.baseballthinkfactory.org/?URL=" prefix? It's now very difficult to make out what a link is by hovering over the linked text, and it also breaks the browser's color scheme for new and visited URLs.

This change was made to discourage spammers from adding links to to comments. It's a growing problem and it requires more of my time than I'd like.

As for the login issue, I have tried to replicate the login problem on the many computers, phones, and tablets at my disposal and can't. Has everyone tried deleting cookies and checked the "Auto-login on future visits" box when logging back in?

This is weird...hot topics says this thread has 102 posts, but I can only see up to 99. Even if I click on the "Last" -- which theoretically would take me to the second page -- I still only see one page and 99 posts. (And the URL correctly has the /P100/ on the end, too.)

When I rewrote the Hot Topics script, I didn't add a check for Open/Closed comments. I also started closing comments instead of deleting them recently. Put the two together...

I will fix the script. Nice catch, BTW.

Thanks to everyone for taking the time to point out these issue. I really appreciate it.

One weird thing I've noticed is that if I click a link on the Hot Topics bar, the link to the comments thread from the article posting itself stays blue, as if I'd never visited the page. The same is true in reverse. Not a huge problem, but I thought I'd bring it up nonetheless.

When the bookmarks come back, will all of our saved bookmarks come back as well? I had a few things tagged on there that I'm not sure I'd be able to find again in the archive search, and I'd hate to lose them.

I'm having the multiple machine problem as well - I logged in from my work computer, then had to re-log in from home. Now I'm back at work, and being asked to log in again. But I noticed over the weekend (when I was home) that I wasn't being asked to log in repeatedly from the same machine.

For what it's worth I no longer have to log in at work, where we use IE7, but still have to log in every day at home where I use IE8. I have deleted cookies and checked the boxes at home but haven't deleted cookies at work in years, nor have I checked the boxes.

As a follow up, I only used my work computer to log in the last 3 days. Everything was "normal". I logged in 3 days ago and have not had to log in any time since then. It's a laptop, so I've brought it from work to home and back and never needed to log in, with multiple re-boots.

Yesterday evening I logged in on my home computer. This morning I needed to log in on my work laptop. So I'm pretty certain that logging in on my home computer caused the loss of my auto-login on my work laptop.

It may be be our problem. I don't use sessions currently, however. I just use cookies. I will look into it at some point tomorrow and will add this fix into my necessary code hacks if it does prove to be the issue.

I am logged in on three different computers in three different browsers. I am also logged in on my phone's browser. Over the last week I've tried logging in at work remotely, on my phone, and at home simultaneously, closed the browsers, was still logged in when I reopened all my browsers on all my devices.

I don't doubt that some of you are experiencing the problem. What I haven't been able to do is replicate it so I can find the cause.

I'm having the login problem, too, just in case you're trying to keep count (chrome at both locations, android phone).

---

Here's a weird one. On the NBA thread right now, the counter doesn't agree with the number of posts because of the 20 or deleted spam posts from the weekend. So right now, the counter shows 2204 posts, but there's really 2100 something. So if I click on 'last' in the hot topics bar, there is no 23rd page to take me to so it takes me to page 1 of the thread. Obviously this is a relatively minor thing, but thought I'd at least mention it.

I wanted to edit that last post, since I'm off a couple of pages (it's really 2404 posts, and 2385 in the thread; so it's the 25th page, but whatever) and wanted to add that clicking 'last' inside the thread (like from page 1) takes me to the actual last page.

Anyway, when I click 'edit' in post 126, it actually takes me to the top of this page and I can't edit the post.

Login problem: Maybe I just didn't notice it before but when I got home, I had to log back in. I am researching things a little more harder now.

Comment editing on this page: fixed.

I am looking into the closed comment count problem. I initially thought it was a problem with my Hot Topics code but, after looking at the code, it's not.

Harold, the nofollow wouldn't have any effect. I hid the member profiles to discourage spammers from creating spam accounts to get link juice from links in member pages, but I'm still getting multiple new spam accounts every day.

Yeah the problem with things like nofollow and Jim's good idea of hiding member profiles is that spammers don't take the time to figure out whether a site is a good target or not. They just hit as many sites as they can.

So, I have to log-in every time I turn on a computer or my Blackberry (BB) or am inactive for 30 minutes or so on the site. This is happening on all my Macs and PCs, at home, at work and on the road, on IE, Safari, Camino, Mozilla/Firefox and my BB Browser. It has happened to me in NY, SF, San Diego and NJ in just the last week.

Close to 100% of my BB posts are being eaten. Often the post disappears and I am taken to a blank post reply screen where all the text is gone. Sometimes if I preview first, the post works. Sometimes it does not. Sometimes, if I respond to a post, it works, and other times, it goes to the blank post reply screen. It is quite frustrating and all new since the, uhm, upgrade.

Handle, I turned off secure forms. Apparently, it was causing you problems. We can continue to get by without it for now.
---
I've turned off Rank Denial for user submitted links for now. As HCO, said the spammers probably don't notice anyway. I will implement Vinay's suggested EE addon for applying nofollow to links, so members will be able to still see links when they hover over them. I will have to come up with another option for the forum links.

I've spent some time going over EE's logon code. They overhauled the logging in process. Some of the changes enhance security; some of them don't have a real purpose. I've made a change to one of the ways EE processes cookies. They were doing something funky that, frankly, makes no sense at all. I'm still poring over the code to see if they are anything else worth amending.

Can people clear their cookies and let me know whether the logging in situation has improved?

I'm testing some changes on this blog, primarily the way the underlying templates are put together. As you can see, I have the basic bookmark stuff done. I'd like to take one more crack at the bookmark Ajax code, though. Bookmarks should be back on the whole site in another day or two.

I have not been able to log in on my iPhone at all in at least a few days. I see the log-in form, enter my info, see a page saying "Thank you! You are now logged in," but it doesn't take -- if I'm trying to access the lounge, I get bounced back to the login page; in the mainland, I go back to my mainland screen, but am still logged out. I've cleared all cookies, history, data, etc., numerous times.

Frigging iPhone. If this was happening on a real browser, I'd tell you what's happening with my cookies, etc. Anybody else still having problems like this? I'm using an iPhone 4S on the latest version of iOS.

Thanks for all the effort lately, especially the debugging. I know it's not fun. Also, thanks for changing the linking back -- I know a lot of people have posted that they're happy about this.

Harold, if you haven't already, go to Settings > Safari and hit Clear Cookies And Data. Yes, this will impact any site for which your phone is already storing log in information. If you'd like to first try a less nuclear option, go to Settings > Safari > Advanced > Website Data and remove the listing for BTF (or just hit the Remove All Website Data button).

I have not been able to log in on my iPhone at all in at least a few days. I see the log-in form, enter my info, see a page saying "Thank you! You are now logged in," but it doesn't take -- if I'm trying to access the lounge, I get bounced back to the login page; in the mainland, I go back to my mainland screen, but am still logged out. I've cleared all cookies, history, data, etc., numerous times.

Frigging iPhone. If this was happening on a real browser, I'd tell you what's happening with my cookies, etc. Anybody else still having problems like this? I'm using an iPhone 4S on the latest version of iOS.

Thanks for all the effort lately, especially the debugging. I know it's not fun. Also, thanks for changing the linking back -- I know a lot of people have posted that they're happy about this.

I had this problem and I finally figured out that in the Settings->Safari the AcceptCookies flag keeps getting set back to "Never Accept Cookies". I had to toggle it again this morning to get it to work for me.

I have the 4S as well, so I appreciate the help from you guys, Harold and Mark. I haven't Lounged from it much, especially since this happened, so if it ever starts working "right", please let us know here. Thanks.

I have the 4S as well, so I appreciate the help from you guys, Harold and Mark. I haven't Lounged from it much, especially since this happened, so if it ever starts working "right", please let us know here. Thanks.

Jim, several of us in the Lounge have noticed that the Search function has been having problems. If a keyword is found in many replies in a thread, upon clicking the "Search this thread" link it gives an error saying that the search did not return any results. And a couple of days ago I searched for a phrase that was present in that day's lounge and that I knew had been used in several previous lounges. That search also didn't return any results. I assume that this is another SNAFU with the latest EE update, since I haven't had either of those problems before. If you could look into this I would greatly appreciate it. Thanks.

fra paolo, sorry. I'm in the middle of restructuring the site's archive structure. If you have the links, you should be able to display them now. Eventually, though, Newsblog entries older than six months will only be accessible in the new site archives. This is being done both for organization and for performance.

Jim — Thanks for the site updates and improvements. Glad to have bookmarks back, etc.

I just noticed the problem mentioned in #100 seems to be back, at least for the Starlin Castro thread. As I'm writing this, the links in the sidebar (Hot Topics and Bookmarks) have the thread at 5195 comments, but the thread actually has 5202 comments. It's not a big deal, but the link takes people to the wrong page when the discrepancy occurs around a page-flip (i.e., last comments are in the high '90s or low '00s). Thanks.

Greg, some of the the changes will be implemented after the move to the new servers and within the new design. I am completely re-factoring the data structure and back-end template code. By doing so, I'll be able to provide many more customization options for members to satisfy the individual needs of as many people as possible.

For example, some people are asking for more off-topic content, like football and basketball. Other people, like you, are asking for the ability to ignore such content. With the re-design, members will be able to choose whether to include those sports or not in the site-wide Hot Topics box by setting the option in their Profile Settings. Additionally, each member will have his own "My BBTF Homepage" (name may change) where he can subscribe to specific content topics by blog (BBTF, Hall of Merit), subject (Red Sox, Steroids), or category (Baseball, Football).

Look for a new Updates post in the next day or so. In it I will discuss some of the upcoming features and ask for a few more Alpha/Beta Testers.

I'm not having that problem, but for a couple of days there's something funny going on with caching. I go to the home page and it's cached from last time. I have to refresh to see the updated Hot Topics. So I refresh the home page and, for example, the Jeter thread says 47 comments. So I click on the link from Hot Topics and I go to the page and it only has 38 comments, the same as earlier today. I refresh the page and see all 47.

So I'm guessing my IE is caching the pages, but the site never behaved like this before.

Additionally, each member will have his own "My BBTF Homepage" (name may change) where he can subscribe to specific content topics by blog (BBTF, Hall of Merit), subject (Red Sox, Steroids), or category (Baseball, Football).

Thanks for the additional information on the caching issue. I have added it to my bug fix list.

Can the people having the IE caching issue please let me know which version of IE they are using?

@Chris and Jose: I more excited about the upcoming changes to the site than at any other time in the site's existence, and I include when Sean and I first released BaseballPrimer.com. My primary focuses on the updates are serving pages as fast as I can, enhancing the community nature of the site, giving users more control over what content they see, helping them find the news they are interested in, and making the site as easy to use as possible.

Thanks for the additional information on the caching issue. I have added it to my bug fix list.

Can the people having the IE caching issue please let me know which version of IE they are using?

- IE 8

My page is still stuck on Feb 14 at about 9 AM CST, which is weird because i wasn't on the computer then. No new posts are showing at all. I tried closing, clearing and restarting and have the same problem

I'm getting logged out over and over again. I couldn't remember where I should post about this sort of thing, so I chose here.

I still can't login on Firefox, which is a pain in the butt because that's what I use for everything else. I've raised this issue several times but still haven't gotten a satisfactory answer, since it's ONLY BTF where I have this problem.

And yes, I've tried enabling cookies, deleting cookies, and enabling cookies so many time I feel like a Pepperidge Farm bakery. Still hasn't solved the problem.