Hello, Dreamwidth! We're ringing in 2012 with a collection of useful information and news from the DW head office (aka my house) and auxiliary offices (aka Mark's house and Fu's house). Behind the cut you will find information on:

* Development, Coding, and Styles* Community and Icon Importing* Site Growth and Stability* Invite Codes* SOPA* The Ada Initiative* Paid Accounts, PayPal, and DW Business* The Great Community Rec-O-Matic

Development, Coding, and Styles

Most of this week's development effort went to more long-running projects (and thus the bugs weren't completely resolved yet and don't appear on this list), but the biggest highlight this week includes finally having resolved that really annoying bug where interest counts weren't being updated properly, leaving us with a number of interests being listed as having 16,777,215 accounts having that interest listed. It was a bear to track down, but thanks to some amazing detective work from a team of people, with heavy lifting done by exor674 on the code end and alierak on the sysadmin end, interest counts should now be a lot closer to reality.

For those who are just joining us: yes, you too can come hack on the DW code! We offer hosted development environments so you don't have to faff around with installing the code on your own development machine (or have your own development machine), and our Bugzilla install is public for those who are looking for a project to pick up. The Getting Started and Development categories on the DW volunteer wiki have useful information on the development process, and dw_dev and dw_dev_training are the communities to go to on DW itself. Or, if you'd just like to browse the code, you can read through the Mercurial repositories; the two repositories that most development happens in are dw-free (for the freely-available, open-source-licensed code that primarily runs the service) and dw-nonfree (for dreamwidth.org-specific code, or things that are part of our site branding or identity).

DW backend development is mostly done in Perl, with a little bit of Javascript on the frontend here and there. If your coding skills lean more towards HTML and CSS (or S2, the custom layout language shared by DW and LJ), you may be interested in creating or converting themes and styles. If you've created a journal layout that you'd like to offer as a system style, you can submit it to dreamscapes. Or, if you'd like to wrangle others' submissions into a form that can be used on the site, you can take the layers and styles that people submit and convert them into a patch that can be applied to the codebase. (There's a guide for people patching styles, which assumes a level of familiarity with both S2 and the process of working with the code and creating patches -- we'll work on getting the process explained in a more beginner-friendly fashion, but you can always ask in dw_dev_training for help.)

Community and Icon Importing

After last week's announcement that community importing was available, mark spent a few days coding his fingers off to improve and optimize the importer to better handle larger import jobs. Thanks to his tireless work, we've been able to lift one of the restrictions we had to place on community import: communities that have over 100,000 comments can now be imported.

If your community has more than a few hundred thousand comments, the comment import may still fail. We're working with the admins of a very large community to work out the last of the problems. We're asking for now that if your community has more than 500,000 comments, please hold off on doing an import for another week or so -- the code won't enforce the limits anymore, but it will definitely make everybody's lives easier.

We also should also be able to lift the restriction that only paid accounts or paid communities can use the community importer in another week or so, once we're fully satisfied with the optimizations we've made -- stay tuned there. Mark's also actively working on the ability to "claim" your posts and comments that were imported and attributed to your OpenID account, so that should be available soon as well.

Meanwhile, we've made a change to the way icon imports are handled. Previously, if you had more icons available on the remote site than you had icon slots on DW, the site would only import your default icon. We ran into some problems this week with LJ reporting the wrong number of icons if you deleted some icons recently (so if you had 102 icons on LJ, then deleted 3 to bring it down under the 100 icon slots available to a paid account on DW, the system was still reporting you had 102 icons and thus the import was erroring). Rather than spend hours and hours of our time troubleshooting, we just lifted the restriction, and now import all the icons you have on the remote site.

As many people have noticed, this is allowing some people to circumvent the restriction on icon slots and use more icons than their account type allows. The reason we don't run the process to deactivate icons immediately upon import is because at that point, we don't have enough data about the imported posts and comments to tell which icons are the most-used icons in the journal. (When deactivating icons because an account expires, the least-used icons are deactivated first.) Rather than just guessing randomly at which icons should be deactivated, we made the deliberate decision to let it slide for now, then clean up in a week or two once the rush to import has died down a bit. (The extra icons won't be deleted; they'll be deactivated, precisely like what happens when a paid account expires.)

If you import your icons and it'll bother you to have icons available now and have them be deactivated later when we do the cleanup, or if you'd like to help us out a bit, you can force a deactivation by visiting the Edit Icons page while logged into your account. This page loads the routine that checks your current icons against your total number of icon slots, so will deactivate (or activate, if for some reason an account upgrade doesn't reactivate inactive icons) icons until you have the right number.

We know this will let some people have access to more icons than their account level allows for a bit, and thank you to those who reported the situation! It really did just work out to save us a lot of time to do it this way, since we had a lot of people who were having import problems and troubleshooting the issue would've taken valuable time away from other projects.

Site Growth and Stability

mark made an excellent post about DW's load vs capacity, and how we're doing in terms of being able to support all the new traffic we've seen this month. The great news is, we're handling the increased traffic wonderfully: we've seen a few bumps along the past few weeks (everybody's watching dw_maintenance for details, right?) but on the whole, things have been ticking along nicely.

In the past few weeks, we've added two new webservers and two new database servers, bringing the total number of machines up to 12: four database machines (since we add dbs in pairs), four machines that work as webservers only, two load balancers (the load balancer + a backup load balancer in case of failure that also functions as a webserver), the search machine (which also functions as a webserver), and the admin machine.

We constantly keep an eye on site performance to see if and when we need more hardware, and since we still rent our dedicated servers rather than colocate our own hardware with a hosting facility, we can get our hosting provider to add more hardware when we need it -- it's more complex than it would be for you to rent webhosting for a small project, but it's not as complex as it would be to buy additional hardware ourselves and have to set it up.

We'd like to move to owning servers and colocating in the future, but for now, dedicated hosting is more cost-effective for us: the way the numbers work out at the moment, it'd take us nearly three years to "earn out" the initial hardware purchase, in terms of amount saved monthly, and there are a lot of practical benefits to dedicated hosting. (Mind you, there are a lot of practical benefits to colocation, as well, but those don't start to show up until you're much bigger than we are right now.)

So, in other words: we're really happy with the site performance right now, and we have a clear plan on how to increase our capacity as our traffic and load increases. mark also has a lot of experience with handling high-availability web applications, and he's very comfortable with how things are running. If you've been worried about our ability to handle this month's increased traffic, you can rest easy.

(Also, for the record, since we name our database clusters after stars and constellations, the new cluster, which is on the new database pair, is the Draconis cluster, joining Alpha Centauri, Betelgeuse, and Capella. If you're curious, you can ask the site Where am I? This has no real bearing on anything -- all of the clusters save for Draconis are on the same physical machine -- but dividing accounts by database cluster even if they're on the same machine makes certain maintenance tasks easier.)

Invite Codes

Speaking of site growth and how happy we are with site performance, many people have noticed that we haven't yet re-enabled the need to have an invite code to create a DW account. We'd intended to leave invite codes off for the second half of December as one of our regular no-accounts-needed account creation phases. After adding the additional servers necessary to support the traffic bump, our performance and capacity was large enough that we decided we could support open account creation for longer.

So, invite codes will definitely be off for a little while longer -- this means that anyone can create a Dreamwidth account without needing an invite code for now. Rather than specify an exact time period for how long invite codes will be off, we've decided to just let open account creation run for an unspecified time, as long as this increased demand keeps up and as long as we're happy with the site performance. (And, of course, as long as the dw_antispam team is able to keep up with the spammers that open account creation inevitably attracts -- all praise be to our dedicated team of spamwhackers!) We'll let you know when we turn invite codes back on.

SOPA

SOPA (the Stop Online Piracy Act) and PIPA (the PROTECT IP Act) are two bills pending in the US legislature to force online service providers (such as Dreamwidth) to remove enormous amounts of content from, and block access to enormous amounts of, the internet, based on the demand of intellectual property rightsholders in an attempt to combat digital intellectual property infringement.

The Electronic Frontier Foundation has a more detailed explanation about why these bills would make bad law: Internet Blacklist Legislation. Dreamwidth joins hundreds of other companies and organizations, both inside the tech industry and out, in standing opposed to these attempts to censor the free and open internet.

We'd like to take a moment to ask you, if you live in the US, to write to your legislators and ask them to oppose the bills.

The Ada Initiative

We said a few weeks ago that we'd be donating 10% of December's gross income to the Ada Initiative, a new nonprofit corporation devoted to supporting women in open technology and culture. (Disclaimer: I volunteer on the board of directors, but I receive no compensation for doing so.)

We're pleased to announce that thanks to a record-breaking month of sales (which we totally did not expect, omg!) we are donating $4625 to help fund the Ada Initiative's 2012 programs. This makes us incredibly happy! Thank you to everyone who bought paid time during December and helped increase that number. Meanwhile, if you'd like to contribute more directly, you can still donate to TAI.

Paid accounts, PayPal, and DW Business

We're obviously very pleased about the aforementioned record-breaking month of sales, and I wanted to say thank you to those who have bought paid time -- it was a fabulous way to close out the year, and looking at the figures on our payment backend has been very, very encouraging lately.

For those who have been wondering what we do with the money from paid accounts: there are two major categories of expenses we have on a recurring basis. The first is professional services; this covers things like processing fees imposed by our payment processor, as well as hosting and backup services. (The processing fees work out to be around 7-9% of all payments, between the flat fees and percentages from our merchant bank account and the flat fees and percentages from our payment gateway; the hosting and backup services run around $4500 a month with the latest upgrades.) The second category is salary: fu is employed by DW fulltime, plus we pay our backup sysadmin and one of our Terms of Service people on a retainer basis as contractors.

We also pay me a very small amount monthly to be working on things full-time; previously mark was also working full-time on DW, but he needed to take a non-DW job because at the time we couldn't pay him what he needed to be earning to support his family. (I'm lucky; my wife has a very good, very stable government job, and we live in a much less expensive area of the country.)

We post an annual end-of-year analysis of our financial position to dw_biz every year, which I will start working on as soon as the end-of-year bookkeeping is finished. Before this month's payments explosion, and aside from the one-time income from our May '11 Seed Account sale (which we did in order to replenish our "war chest", the money we keep in reserve against future disaster, after nearly three months of being unable to accept payments in early 2011), we were already on target to take in slightly more than we spent in 2011. (Which was incredibly exciting, let me tell you!)

This past month's increase in income has allowed us to increase our server power, continue to restock the war chest, and invest in some infrastructure upgrades. (Plus, we bought mark a pizza while he was hacking on community import.) Beyond that, we don't want to make long-term plans counting on this level of income as a given, but we're really hoping that if this level of support continues we'll be able to get mark back on DW work full-time and be able to pay me market rate for my full-time work on DW.

The Great Community Rec-O-Matic

We've done this before, but given how many people are just joining us, it's time to run another round of the Great Community Rec-O-Matic.

Here's how it works:

* If you're looking for new communities to join, comment here with a list of some of your interests, and the kind of community you enjoy reading and participating in.

* Or, read through the other comments and see what things other people are listing. If you see someone you think would be a perfect match for a community you admin or participate in, comment back to them and point them at the community!

*

That's it from us for another update! As always, if you're having problems with Dreamwidth, Support can help you; for notices of site problems and downtime, check the Twitter status page; if you've got an idea to make the site better, you can make a suggestion.

(Also, I am very close to having followed the hobbit tradition of giving presents to other people on my birthday -- if I'd started this a day earlier I could've posted it on the 5th. Thanks to everybody who sent me birthday wishes!)

We (and my runaway abuse of parentheses) will see you next week for our next update!

no subject

(Also, I am very close to having followed the hobbit tradition of giving presents to other people on my birthday -- if I'd started this a day earlier I could've posted it on the 5th. Thanks to everybody who sent me birthday wishes!)

no subject

Right, shall try to make use of the Rec-O-Matic 8DI'm looking for- Kaizoku Sentai Gokaiger- Prince of Tennis- Super Junior- miss ASo if anyone has fanworks and/or discussion places for these at hand it'd be awesome if you could point me there :o

no subject

I can't express enough how impressed I am at the yeoman's work y'all have been doing. Despite what must be a MASSIVE load of imported stuff, DW has never been inaccessible to me the entire time (oh it's loaded slowly, but that's different!)

I also want to thank you for making a stand against SOPA/PIPA. Is there a DW comm that has taken up analyzing it? If so, let me know, because the implications are terrifying.

Betelgeuse, Betelgeuse, Betelgeuse!

Am totally looking for communities of people talking about library and information science issues - not necessarily limited to traditional libraries! Basically people who geek out about things like information organization and presentation, knowledge management, taxonomies and ontologies, the user experience, etc. I do know about librarians, though it's been a bit quiet there - if there aren't other library-related comms out there, perhaps we can rejuvenate this one?

And for residents (past, present and future) of Cambridge and Somerville (MA), I offer up camberville. Let your neighbors know about fun things going on! Reminisce about Bartley's Burgers and how much better it is to watch the fireworks on the north side of the river! Kvetch about the dratted Red Line shuttles! Plot in-person Dreamwidth meetups! (My own pledge to contribute more frequently: I will regularly round up all of the new hyperlocal restaurant and food truck openings that I hear of. :) )

Communities!

These are my general recommendations for anybody who's looking for things to read and do and thinks the topic sounds interesting:

followfriday, which is the DW version of the Twitter phenomenon, and sometimes has awesome recommendations. (Seems to have slowed down some.) omnomnom, for delicious cooking and baking. bitesizedcleaning, for tackling large and recurring housecleaning adventures in steps much easier to sustain than "Clean ALL the things!" bestthingever, for pan-fannish reccing. sauce, for general reccing of awesome things. capslock_dreamwidth, FOR ALL-CAPS HILARITY AND YELLING. fandomcalendar, for every fannish and vaguely related event members can think of. metaquotes, for funny things across the site (a little slow, but good).

no subject

And I'm so pleased to work out my credit card works here; I held off buying time here thanks to the lack of PayPal (although I utterly understand why you would reject using it, and kudos for standing up to your principles), but I have a credit card, and it works, and all the paid time can be mine! All the pretty icons! And, oh, yeah, supporting you guys :D

Thank you very much for these regular updates - they are useful and informative and lovely.

Recommendation if you like choir music

Lately I've been reccing choralmusic, because it would be great if we had more members. So if you like music performed by choirs or sing in a choir yourself, come and talk about rehearsing, performing, awesome pieces and choirs or recordings you admire.

no subject

WHAT DO I LIKE LET'S SEE- Super Sentai- Kamen Rider- Roleplaying... and that is much of what involves me lately. :|a

AS FOR RECCING, I REC cfud which is a very fun rping game and we have a very large cast of Super Sentai and Kamen Rider characters if anyone wants to stalk/join us/what have you! Man, just be my friend if you like that stuff.

no subject

It still freaks me out that there are regular updates. I mean, I'm used to hearing crickets on LJ, which was all fine and dandy...until now. These updates make me grin like a loon, which admittedly scares the local wildlife, but hey, sacrifices must be made! :D

I have one question: Will there ever be additional icon slots we can buy? I know that 100 icons is a lot, but I'm...kinda used to having my 250+ icons available. *facepalm* I think I'd flail and squee for three days straight if DW ever decided to offer extra icon packages... *hopeful puppy dog eyes*

Anyway. Happy belated birthday and Happy New Year and yay, comm importing! \o/

no subject

It's definitely on the list of things we want to do, and ideally we'll let people decide how many icon slots they want to buy at a time (instead of offering them in certain packages, I mean), up to a certain point at least. It's a huge project, coding-wise, though -- LJ's entire payment system isn't open source, so we can't use any of their payment-related code, including the icon packages -- and the way we want to do it, which is different than how LJ does it, is going to involve a lot of rethinking the whole concept of how payments and user capabilities work (which was first set in 1999 or so and isn't as flexible as we'd like it to be, etc).

I should note, though, that premium paid accounts do get 250 icon slots (compared to the 100 icon slots with a paid account), so there is a way to get more icons than just the 100!

rec-o-matic

Are there any comms for Xenosaga, Yū Yū Hakusho, and Final Fantasy VII? I'd love to know what other Naruto-based/focused comms there are around here too. (I'm a member of kakasaku but I don't know of any others.)

no subject

Heee, I'm on the Alpha Centauri cluster. For some reason, that fills me with glee! *g*

Re: SOPA. Is there anything that non-US citizens can do about it? Cos omg, it's terrifying.

I'm looking for an active comm about Kdramas. Also, one about books. Maybe a reading book club or a comm where people rec books? Gah, now that there's the Community Rec-O-Matic, I can't think of anything else. It figures!