Siddartha has asked for the
wisdom of the Perl Monks concerning the following question:

I'm not sure if this is the right place to ask my question, but here goes:

I have been asked to develop a website for a company.
It will basically be a jobsite where users can register, upload CV's etc.
and Employers can post jobs that are searchable by different criteria.

I don't have the detailed spec yet, but need to know how to estimate
a delivery time.

This project will be done after hours, since I am working full time.
I need to figure out how much time/day I can commit and how long it
would take to complete.

I have decided that the cheapest and best option would be to use
Perl and MySQL, hosted by an ISP.

Because it is hosted by an ISP, I don't have control regarding the installed
Perl modules. It would have liked to use Mason or something similar
using mod_perl but I'm not sure if it is, or will be installed.

I am not a Perl guru at all. I have only been using it up to now for basic
CGI scripting. I have used MySQL before, but it is going to take me
a while to define the database structure and the interfacing.

Any comments on calculating development time, or possible ways
of implemeting something like this would be really appreciated.

I don't have the detailed spec yet, but need to know how to estimate a delivery time.

You can't. Any number you give the
powers that be will be bogus.

Them: "I need you to drive me somewhere. How long will it take?"
You: "Well, where are we going?"
Them: "Oh, just out of state."
You: (thinking... "Hmm, not detailed, but I have an idea.")
"Oh, maybe a day."

...later...

Them: "OK, we're driving from New York to San Diego.
And we'll be there tomorrow, right? 'cause if we're not,
it's your ass."
You: "Uh-oh."

I have decided that the cheapest and best option
would be to use Perl and MySQL, hosted by an ISP.

How could you have made that determination
without a detailed spec?

You: "We'll be making this trip in my Yugo."
Them: "Oh, by the way, I have a few boxes of stuff I have to
bring. Well, OK, a lot of boxes of stuff. And a refrigerator."
You: "Uh oh."
Them: "And we'll be there tomorrow, right?"

The end result?

Them, to others: "Man, all I wanted was a little trip,
and he just screwed the whole thing up, and it took forever
to get there. I wouldn't use him again if you paid me."

Closely related to the Scotty principle; tell them 3 times longer than it's going to take, then finish it early and say "yeah, things slipped into place a little sooner than expected". Trick to making this work: ensure that you end up charging less, too.

I won't try to answer all of the issues here, because it would take all day.
Your first problem is here.

I don't have the detailed spec yet, but need to know how to estimate a delivery time.

If you don't have a detailed spec, it will be difficult to estimate the delivery time.

Start by deciding how much work needs to go into producing the detailed spec.

You may want to agree that development as a set of milestones, e.g. :

Rough Specification

Detailed Specification

Initial Design

Detailed Design

Initial Development

Revise specifications

Final Development

You may only be able to give rough estimates for more than the first few, but you should be able to refine the estimates as you get further into the project.

Clearly you need to know the capabilities of the hosting platform so that you know what tools you will be able to use. If you have to build some functionality from scrath, it will obviously take longer than if you can uise existing tools.

One way to get ideas is to find a list of existing sites and get the client to give you feedback as to whether they like some of the features, or how they would like it to be different. That can often give you a more concrete feel for the detailed requirements.

These are just a few thoughts. You really need to work up a detailed list of questions to make sure that when you start, your expectations about what you are going to deliver are the same as the client's.
--Brovnik

From Murphy's laws for programming:
If something is easy, it'll take only twice as much as you thought. If it is difficult, it'll take at least ten times as much. : )

No jokes now, project time estimate is a serious matter.

I agree with the steps above, and I offer useful 'trick': Make customer aware that his/her thinking time also needs to be included, too. I.e. "I'll need 2 weeks to develop Detailed Specifications *after you finished* reviewing and we agreed on Rough Specs."

Of course you will try to start 'Detailed Specs' right after finishing 'Rough', without waiting, but you need to won time… Like in chess game, to won you need to use your opponent’s time to think about your own next step.

Obviously if you thought you need about 2 weeks to do something and they wasted 10 days reading specifications and responding to your questions, you still need 2 weeks to finish the job, right?

Also, take into account you need some time in the evening to ‘recall’ where you finished last time. I call it ‘time for swap’. And like with disk swap, the more often you swap, the less time is available for a useful work. Looks like a lot of weekend work to me.

You may also ask questions:

How complex is the project? Does it involve new skils? Do you want to learn these skills for free, or you do not care?

Is it for the same employer you are working daytime? Same manager, same computer, or other? It might be bad.

Do you have good relations with 'night project' customer? Afraid to stain them?

Who will be available to answer to night project questions, and when?

Who will decide quality of project completion? Penalty for missed deadline

Is anybody else involved in the implementation? If yes, how much control do you have over other developers?

I read a horror story about case when guy agreed to program in night something for his daytime company, was asked to take another guy to participate, then the other guy got behind schedule. To make for missing time he started sneaking into ‘night project’ during the day, and his day manager was not excited, because he wanted him to work overtime for day project, what was also behind schedule. As it happens, specifications were changed a little, he missed deadline a little, was required by his night manager to accept penalty…

As a result, he left company with bad feelings with his day time colleagues and manager, and was not paid fair price.

I think this order of tasks is pretty good (especially because it "plans to throw one away") but incomplete. People (even technical managers) often completely omit testing time from project estimates. It is as if once the code is written the customer can deploy that afternoon.

I can see some naive justifications for this, but I have experience to the contrary:

Testing is done during development so it does not need to be scheduled.

Developers rarely test their own code this thoroughly, especially when they are on a deadline. Even when they do they inevitably miss something or other that's pretty serious.

Testing should not be scheduled because no one knows (or can know) how long it will take ahead of time.

That does not make testing instantaneous. Fred Brooks, author of The Mythical Man-Month (which is published in book form along with other essays of his; I strongly recommend the book) advises that a full one half of project timelines are typically spent on testing, whether testing time is scheduled or not. I have observed this myself to be a pretty good heuristic.

Another mistake people make is not having a test plan, or if they do, not scheduling test plan development! I have found that a test plan can be derived from a thorough design document fairly easily.

Testing should not be scheduled because the customer will get mad and possibly go with another bid.

The best of the three argumento IMO. Try to explain to the customer that testing is vital and that other vendors will spend just as much time on it; it's just that *you* are being up front and honest about it. If that doesn't work try not specifying a delivery date due to indeterminate testing time. Last resort, put an "I told you so" clause in the signoff document.

not having telnet access can be a pain when using perl scripts -- the php
mod loaded into apache is faster, and php is like a cross between perl and c.
I'd recommend using that for what you are trying to do. telnet access is the killer.
WP

Yes these formulae are silly but experience seems to bear out these basic principles.

The first step is to define your project in precise terms, what do you want
your site to do, why, and how...This is vital and will save untold hours in the long run.

The answer then depends of course on your knowledge and experience, your spec, and how
little you will actually have to code by adapting available modules/code to
your needs.

One problem with modules is that they *can* have a steep learning curve, so
you should possibly stick with the basics. CGI.pm will be essential.

One approach is to try to find a script that fulfills most of your criteria and
then adapt it. Sadly you tend to get what you pay for but of course there are so many
exceptions to this rule...especially in the give and let live Perl community.

CGI.pm is the basis from which to start. It can generate HTML for you as well
as get CGI input. It allows file upload so users could upload say HTML CVs.

Search engines to index your data can be complex to code. In a pinch you might
consider one of the free site search engines. I use http://siteLevel.whatUseek.com
for a cheap and cheerful search, on shoestring sites. You can customise the search interface and the output
template and it is free for ?1000 pages I think. To see simple example in use check out
http://www.fg3.com.au/home.htm. This site (still under construction of course!) took 6 hours to generate, graphics, search, etc.
Try the search, and then look at the 4 lines of source HTML that are
required. If you have CVs and Job Ads as *.html pages that can be spidered this will solve
the search requirement in under an hour. Downside -> ads on result page. Hint, edit the template
and dump the adds to the bottom - if you remove them they get replaced at the top, but if you
move them to the bottom all is OK.

Consider employing a perl consultant to set up at least the basics for you to tweak

It should not be a problem to get modules installed at your ISP, often in your local bin
where you may or may not be able to tweak them. When it comes to CGI most ISPs are aware of the
security risks and have varying policies. Ideally you will want to be able to
freely upload your script during testing but this requires that your ISP trust
your code, or be non security concious. This is a real issue when it comes down
to getting your script running. If you have to get your script vetted (often for a fee!) with each
change the process becomes very slow. This *should* influence your choice of ISP. PS You may not want
your prodution code to reside on an ISP that let's anyone edit their CGIs but.....

Finally (whew got a bit carried away here) if you want to simulate CGI, have relatively little experience,
are familiar with win32 then perlbuilder 2 from solution soft is a very useful program. Trial copy at
www.solutionsoft.com.

Good luck, I'm sure you will have fun discovering the power of Perl, the greatest CGI lang on earth.
If you have specific code problems that the Monks can sink their teeth into you will often
find help here. Homework and writing your code for you is a lottery and you know what the odds are in lotteries!

I know it's a very general question, and impossible to answer, but you are monks after all. I really appreciate the speed of replies.

I have not decided on an ISP yet, but are looking for something in the UK with full CGI and MySQL support. at the moment Internetters seems like a good bet.
I would appreciate other possible ISP's. I have worked with Internetters before, so that is probably why I would go with them. They hosted a site which I developed for my day job and it seems to be OK.

My biggest problems are:
1. I only have FTP access, no Telnet.
2. I don't have access to the Apache error files

Up till now it hasn't been the biggest problems since I developed everything on a development machine and tested it before I uploaded it. The development and hosting environments aren't exactly the same though, which creates a problem now and then.

I am however a bit worried about MySQL development on a remote host with no error logs. Is there any way I can see what is not working without the ubiquitous Internal Server Error message?

My next question regards searching. If I want to search through the MySQL DB, is it going to be as easy as:

SELECT * FROM jobs WHERE creation_date < $search_max_date

or do I need something like siteLevel.whatUseek.com that you suggested?

It's been a while since I did any SQL, but I'm sure that it's not going to be the biggest problem.

Here are a list of what I think needs to be done: (My solution in brackets, comments would be appreciated.)

I am however a bit worried about MySQL development on a remote host with no error logs. Is there any way I can see what is not working without the ubiquitous Internal Server Error message?

Yes; use CGI::Carp qw/fatalsToBrowser/;

This prints nice error messages to the browser; however, if your script dies before compiling, you just get a (fairly) unhelpful "compile error."

If you don't have a command line, maybe you can do your developing on another box somewhere and then move everything to your host? I personally could not bear to write any decently sized script without the debugger.

i would wholeheartedly recommend Positive Internet (www.positive-internet.com) if you are looking for an excellent Perl and Mysql hosting company in the UK.

Having used a more well known host before Positive, i can state that the folk there are v helpful and the service is outstanding - they have even installed perl modules that they didn't have but that i wanted to use, which has been great.

Tell them iain sent you if you go with them - they have a nice referral scheme too ;)

also, i can't seem to locate an email address for you if you are looking for help via mail - anyone help?

Don't assume that when you do several times the same thing you'll do it faster(In fact you'll probably get bored and it will balance the speed gained by experience)

No real task should be made professionnaly in less than half a day (Real analysis,Test, Documentation...Not only implementation)

Try to estimate the time to do each small task, sum up, then multiply by 2
(test, client interaction, integration...)

If I don't have total control I can't produce correct estimation
(state it but try to give an estimation anyway :
"Provided the ISP had mod_perl properly configured it will take X hours, add X hours if I have to only use PHP...")

Any addition to the specs will change the estimation(even if they take something out, beccause your prior analysis and possibly data structure may be now inadequate...)

Following those guidelines have saved me several time.
"Only Bad Coders Code Badly In Perl" (OBC2BIP)

Try to estimate the time to do each small task, sum up, then multiply by 2 (test, client interaction, integration...)

You can get better at that factor. Keep a backlog of
your estimates and the actual time you actually needed.
This way, you can adjust the multiplication factor for
future estimates. The factor will vary with many influences, including working environment, project scope,
growing experience, business domain, quality requirements,
and so on ad infinitum. Still, you will get much better
estimates by recording your past performance, and extrapolating from that by way of an adjusted multiplication
factor.

I just wanted to emphazize that whatever the time you estimate,
increase it to be able to handle all the possible problems which may arise.
(Furthermore I've whitnessed that almost everybody tend to be too optimistic when estimating...)

You won't lost a penny if you are thru before the deadline.
But being late is always VERY unpleasant.

If you are going to work on it only a few hours a day, I find that a huge time waster for me is getting back up to speed where I left off. (Often I'll work a day or two straight on something complicated otherwise I waste half my time trying to remember where I was).

As far as calculating time (and cost), the job detail is pretty vague. Sort of like asking "How much is a car", there's a whole spectrum from the Yugo to the Bentley. I'd be very careful to state everything that you are and are not contractually obligated to perform and guarantee.

When I first started working for myself, I played it fast and loose. I've done a lot of different types of projects and my <nobr>"Guesstimates" ™</nobr> have always been really close... until.. I had a client that expected our 5 figure software to do everything that there aborted 7 figure software couldn't do. Do to the lack of a rigid spec, it was a painful ordeal. Luckily they had a burn rate on the order of a super nova so the problem resolved itself.

I know this doesn't really help come up with an estimate but I hope it gives you a bit to think about.

Whenever I am quoting a job, I mentally map it out into the base components and actions and think can I do this easily? Have I done it before? If the answer to the latter is no, I do a proof of concept real quick to determine the complexity.

When all else fails, come up with a reasonable and realistic guess of cost and time frame and then double it. :)

I have decided that the cheapest and best option would be to use Perl and MySQL, hosted by an ISP.

I have a serious problem with that. I won't argue the cheapest
part, but if I were to give you the assignment and you came
back to me saying Perl, MySQL, hosted by an ISP in such a way
you cannot control which modules are available is the best
solution, I'd thank you and hand over the assignment to someone
else.

First, let's make the assumption the company actually cares
about the website. If not, well, who cares? But if they
really care, do you really believe it's in the best interest
of the company to host in on an ISP that gives you very limited
control? I seriously doubt so.

Secondly, let's assume the company not only cares for the
website, but also about the data. If you care about data,
MySQL should not even *be* an option, let alone a "best"
option. Check for instance http://openacs.org/philosophy/why-not-mysql.html
for some reasons why MySQL should not be an option.

I will seriously consider other options, if I knew what they are. The client is a startup company, so it doesn't have a huge bunch of capital to spend on the site. They also consider advertising to be the one thing to spend most of their money on, not the development of the site. Obviously the site is important to them, or they wont spend such a lot of their startup capital on it. The site will basically be their means of income, so it is a crucial part of the company.

So, why did I decide on Perl and MySQL?

Firstly because I know Perl (a bit at least, and I really don't want to use ASP), and have used MySQL before for other stuff.
It is the cheapest option.
I don't think the client can afford a dedicated server at all.
That basically leaves an ISP.
If anyone knows of a ISP that is not too expensive and lets you install Perl modules etc., then please tell me.
Regarding MySQL: I don't think that the posting of a job on a jobsite is the kind of data that warrants using Oracle etc. I will not use MySQL at all for billing purposes, at the moment I am looking at using an external service like NetBanx, but would appreciate other options.
I am basically looking for a Database that will store data that I can search. A flat file is too much hassle to use. I do think that MySQL is therefor the best option for me.
If however I find an ISP that has another Database and is not too expensive, I would consider it.

I would appreciate it if you can give me concrete reasons why Perl and MySQL would not be a good solution, even if it is hosted on an ISP.

Firstly because I know Perl (a bit at least, and I really don't want to use ASP), and have used MySQL
before for other stuff.
It is the cheapest option.

That's not much of an argument, is it? Sure, you don't pay
license fees for Perl nor for MySQL. But you don't pay license
fees for Java, C or Python either. Nor for Postgres. Note,
I am not advocating you should use any of those. But "I have
used X and Y, and I don't want to use Z" doesn't make X and Y
the cheapest solutions. Development, maintenance, modifications
all take time - and people working on them will want to get
paid for it. However, I wasn't argueing price.

I don't think the client can afford a dedicated server at all.
That basically leaves an ISP.

Interesting. You don't even know whether the client can afford
a dedicated server, yet you conclude using an ISP is the best
option? Sorry to say, but that doesn't sound like you've done
much of an investigation. However, there's nothing wrong with
using an ISP. But it is wrong to tell your client his best
option is to use an ISP that doesn't give you much freedom.
If you can't even install modules of your choice, your client
is much better off with an ISP that does give your client at
least that.

I don't think that the posting of a job on a jobsite is the kind of data that warrants
using Oracle etc.

Did you ask your client how important they think the data is?
Can they afford losing the respect of their customers (the
people using the website)? If your client doesn't care about
losing the data (resumes, etc), then why are they collecting
it in the first place? Is your client planning to make money
(directly, or indirectly) with said data?

I would appreciate it if you can give me concrete reasons why Perl and MySQL would not be a good
solution, even if it is hosted on an ISP.

I didn't say I objected to using Perl. There might be reasons
not to use Perl, but your article didn't say (if the application
is going to be developed and maintained by a team of 40 Java
jocks with no Perl experience, Perl is likely not to be a good
choice). I did, however, post a link to a page giving a
multitude of reasons not to use MySQL. If those aren't concrete,
what reasons are?

The client is a startup company, so it doesn't have a huge bunch of capital to spend on the site. They also consider advertising to be the one thing to spend most of their money on, not the development of the site. Obviously the site is important to them, or they wont spend such a lot of their startup capital on it. The site will basically be their means of income, so it is a crucial part of the company.

Company decided to be cheap on basic mean of income, and instead to spent most of money on marketing? Looks like "get-rich-quick" scheme to me.

If you decide to go for it, at least get agreement to be paid step-by-step. Protect yourself before possibitity that company will go belly up before you got paid.

For me is very hard to imagine a situation where I can use database without solid atomic transactions and rollback. If MySQL cannot support rollback, I will avoid using it for important production data even if paid to do so. And in no case for payment & Credit Card processing. Are you looking for troubles?

You are taking rather big risk, I hope at least your award is worth it. :)

I've used MySQL for websites that get millions of page views involving up to hundreds of millions of queries per month without ever having a problem.

I should repeat that for emphasis: I've never had a problem. No lost data. No mysterious crashes. Nothing except good performance and reliability. (Performance was about 9 times faster than Oracle by my benchmarks in the application I was running at the time it was ported to Oracle.)

The link that questioned MySQL as an adequate database and questioned whether it was just waiting to fall apart at the seams because it doesn't have atomicity or rollback obviously was put together by someone totally unfamiliar with its use in a production environment and who seems like they were more concerned with communicating their knowledge of esoteric (but good to know!) database concepts than reality.

Of course, it doesn't have as many features as Oracle or other large commercial databases, but it sounds like for your needs, it's just what you need.

After reading this reply, and being a mySQL user, I naturally went to read much of the refered to page (it's very long).

It's a very interesting sequence of posts and responses, not so much because it was in any way conclusive on the "Not MySQL" front but rather because its a good lesson in how things change. It's a conversation that started a year ago and still continues, and shifts subtly as the various systems it discusses have changed over that time. It's also a good lesson in how inteligent people can have radically differing opinions because of the the questions they think are important to ask, and the problems they think are important to solve. I highly recommend it, but again not because it "proves" that MySQL shouldn't be an option.

Here's my personnal experience. I have developed two years
ago a similar site. I was student, perl newbie,
apache intermediate, SQL newbie, XML very-newbie, good
knowledge of linux etc...

I had some technical knowledge in SQL, perl and so on, but no
concrete experience. My colleague and friend (student too)
was at the same level than mine.

We've spent 3 months full time plus 2 months coding during the night. And we went with a mod_perl (Apache::Registry)
site, XML & SQL data, Template system (HTML::Template)

This site was a rewriting of an older one (mod_cgi + msql
+ print $html; + no cookie + no session :)))
Finally, the site major features were : Job posting, Resume posting,
candidature system, application letter, Pro directory ...

I hope this help you to figure out the developement time,
but I guess that it depends on plenty of details... So the other
posts should be more precious than this one :)
Have Fun!!

The other posts does give me a better idea of what to do, but it is really nice to get some actual concrete ttimes. I do really hope that I can do it quicker than you did.

I am doing it on my own (Mad, I know :)). I know CGI scripting quite well, even though I have not used MySQL with Perl at all. I know basic SQL query syntax, enough to update, delete and select data from tables. I don't know if I'm going to use XML at all, not yet anyway. I know a lot about linux, but unfortunately only have FTP access to the site.

I am writing this from scratch.

I have about 4 months to do it, and can only work on it part time. I basically need to know if I am going to crash and burn?

I would really appreciate any specific comments on what to look out for when developing the site.

It will be a very basic system to start of with;
Job posting, resume posting (both as Form data), and search for current jobs/employees. My biggest problem at the moment is that he wants credit card billing for Employers when they post jobs, and I have no idea how I am going to do this. At the moment I am thinking of using someone like NetBanx to handle this for me.

I wouldn't take credit cards unless you can control the
server. Taking credit cards is a big responsibility,
and you will have a hard time proving that you did it
correctly should anything go wrong. Although I am used
to US liability laws (and am therefore paranoid about such
things), I wouldn't want to be the one who got sued because
someone hacked into the ISP and set up a credit card
sniffer.

I set up a credit card system and found it to be painful,
because I had to deal with:

Learning how credit cards really work. This isn't as
simple as you might think. There are many issues that I can
describe if people are interested.

Dealing with the US Department of Commerce denial list.
They keep a list of bad guys that I wasn't allowed to
sell to.

Security reviews by a guy who was long on paranoia and
short on real knowledge. He
insisted that I implement a really twisted system that
had performance problems.

An antiquated order processing system in the company.

There were many more...

Once the site is running, you need one or more people in
a production and support role. This is the area that
many people don't think about at all, and it makes many
sites fail quickly after deployment.

Given the level of experience you have described,
the tools that you have available, and your time frame,
it sounds like a crash-and-burn to me. Debugging problems
will be extremely difficult, especially with your
credit card system, which is difficult to prototype.
For example, say your credit card transaction times out.
How will you solve the problem?

I would /msg this but it's going to be too long, sorry everyone for this ad-space!!!

Apart from Netbanx, you could also try WorldPay. They claim you can get set up with them in around 48 hours - (in reality it takes a little longer because of form filling). There are the following benefits:

You have the option to accept payment in multiple currencies

It's very easy to integrate, although the look and feel is not very customisable.

You don't ever store any credit card information so you don't have to worry about cc info being stolen from your server.

There are loads more options in the UK for credit card billing such as securetrading - but that is much more work to integrate and get set up with them.

On another note, you'll need to decide about who owns the intellectual property, copyrights, etc. Plus, make sure you don't give them the site without getting paid. If this is your first job, then you might fall into this trap - because it's really easy to think that you need to be very accomodating to the client without covering yourself.

A client of mine developed a job site for someone and has only received about 10% of the fee even though the site was finished about 9 months ago.

Bottom line: try and get everything in writing, if possible arange for some payment up-front or after the first phase is ready for the clients to see.

In fact, I think that a very basic system for
beginning could be developed quickly. As I said, my site
was an enhancement of an older one, so we have spent about
one month wondering how we could add the required bunch
of features to the older site. (In fact, the resultant site
was quite complex, featurefull...)
So, you should gain one month :)

You should gain more than one month since we had to learn
the usage of CVS, XML::Parser and how to use Apache::Registry.
More important, we had to find these tools. It's
not obvious when you're just-a-newbie, and you don't know
how-to-do and why =)

Nevertheless, it takes time. A lot of time, since there are
always problems, and your boss/client always add some feature
enhancement... (that's why it's important to have good written specs
and a good contract, as it has been said in other posts)

Here's another note from personal experience. I've also helped developed a similiar system, using Perl and Postgres. I think our project was toward the small end. The features included:

Users could set up accounts and update their personal information.

Admins could add an update jobs and basic organizational information

Public could browse through jobs by category.

The site was small enough that we didn't need a search engine, or mod_perl, or XML stuff. There were 12 fixed organizations that could post, so we didn't much interface to manage that end. I believe we spent about 50 billable hours on that. (factor in that we already have strong Perl and SQL skills and have full control and access to the hosting environment.)

I agree with all the other folks that getting a clear spec with as many details as possible before you start is a Good Thing. We usually develop a functional mockup first, using static HTML and JavaScript clickthroughs on the forms. When that is refined with the client then we use that as the core of the final spec, and make a refined estimate with it.

Don't do too much work before you get a contract signed, because without a contract it can be harder to get paid when things change. In my case, I put in about 80 hours of work (including contract preparation) and then found out that the reason they were so slow getting the signed contract back to me was that they'd hired someone in-house to do the job I was contracting for!

If you have to do a lot of work up front to estimate (which you may have to), try to get paid something up front.

And be careful that you specify the system in terms of behavior rather than in terms of any particular technology (because you don't want to commit yourself right now to a specific implementation).

Development time is one of those bits that you're almost always going to get wrong, as development almost always takes long than you think. For a good read on the topic, check out Fred Brook's Mythical Man Month. It's been around for over twenty years, and is still considered one of the best books on the topic.

With regards to mod_perl, it's unlikely that you'll be able to use it on a shared hosting environment. You would need access to apache's configuration file, for one thing, which would give access to all the virtual hosts on the machine being used by other customers.

This is not so much a comment on calculating development time, as many of the other monks gave considered responses obviously borne from experience.

This is a comment about your ISP.

I would be more aggressive in finding an environment that is suitable to you. What kind of work can you do if you are constrained to a machine you have no control over? (FTP but no telnet? Noone should work under those conditions!) You need to find something else.

I would recommend something like rackspace.com that will simply set up a cheap Cobalt for you (rental if you can't afford to buy one) in a colocation facility with Linux, Perl, Apache/mod_perl, MySQL, or whatever, then give you root.

They charge by the month, so if the project is worthwhile, you'll be able to pay your bills. If not, you just move on.

I would guess that what you pay over hosting off your ISP will be more than made up in the time and cost savings you get by having an environment you've got some control over. And you'll be a much more sane individual at the end of it all.

And if your client can't afford to provide you with at least an adequate and production environment, then they shouldn't be undertaking the project in the first place! If they aren't bright enough or experienced enough to realize this, you need to communicate it to them. It will save you both alot of heartache.

One thing I haven't heard anybody discuss here, which is my favorite method of calculating development time, is to actually just start doing the work, and keeping track of the time you spend, before a contract, before anything. Then, after some period of time, look back on how much you've accomplished think about what's left to do, and you'll be able to offer a pretty good development time estimate.

Call me crazy, but this method works on lots of levels:

Most of us are in this business because we have fun coding, so the time you spend (even if you don't get paid, or have to throw it all away because design decsions change) will have been fun.

You learn tons each time you just start doing the work. I've written complete look-alikes of things like HTML::Template before I knew it existed. Of course it looks like the folks who wrote that did a much better job than I did. So what? Now know exactly what it needs to do, and I'm in a great position to evaluate it and to modifiy it.

It's probably the method by which you'll get the best actual measurement of how long it's going to take. For smaller projects, you get it 100% right, and you bypass the "design specs" section all together. How many of you have had the experience that your clients only half know what they want, and part of what they are hiring you to do is tell them what they want? What better design spec is there than a working project? As long as you have good coding practices modifiying things to the final design (which is never the same as any "design specs" anyways) isn't that hard.

My direct response to you Siddartha is, unless your mortgage and kids are on the line and you won't like what it does to your life-style, then just jump in and do it. Damn the torpedoes. Definately do use CGI.pm, and probably use CGI::Application and HTML::Template these will both save you lots of time in the long run despite modest learning curve issues. Whether you use MySQL or PostGress or flat files, big deal, just use the perl DBI and switch later once you get things up and running.

While everyone else seems to have expressed all the requisite concerns about estimating without specs, as well as some pitfalls to watch out for when estimating, I have an idea on an ISP that might be useful.

Bearing in mind that I have not used them, but merely came across them accidently and read much of the info on their site, (and therefore can't wholeheartedly recommend them) take a look at PowWeb, as they provide web hosting with cgi access, Perl and PHP scripting, MySQL, and a bunch of other things that you don't often find on low-maintenance cheap ISP hosting services. Let me know if you end up using them or not, and why. I'm curious about their absurdly low price for what they offer.

I just sent them an email asking what modules they have available for Perl, if there is a way to request additions, and if they support mod_perl in some fashion. I'll reply to this post if I find anything out.

I am offering an entirely different spin. Since this is going to be a MySQL based jobsite, why reinvent the wheel? There is a wonderful software package called ForwardSQL which is essentially a suite of scripts written in C that I'm sure can do everything you need.
I would buy the package and pass the cost along to the client. Your time will be spent designing the site and NOT writing code.
The URL is dbwww.com. I have no affiliation with them, but offer this suggestion as I have used this software in the past and it is quality code.
You may need to write some custom apps depending on the nature of the project, but by and large the ForwardSQL program can handle 99% of your needs.
I have discovered that by using their code, my development time is quite rapid and has allowed me to win a number of project bids because of the decreased development time.
Just my $mytake=$thoughts*.02; $opinion=sprintf("%.2f",$mytake);

I am seriously looking into this. The only problem will be that I will need a dedicated host, or an ISP that gives me telnet access. The cost of the solution is also not that much, and the increase in productivity should more than make up for it.

Thanks again, I have received a lot of usefull information from everyone, but this is probably the most usefull reply to my specific needs.

Hi, I wrote a pretty detailed response but Netscape crashed.. so here are the highlights.

- Mysql is fine for this job, it will take longer for
Oracle which is overkill. I've used both and Abigail's
comments notwithstanding, I can tell you your choice is fine. I read most of the document she provided and while
there are good points there are also a lot of people with
great experiences on MySQL and I find much of it outdated FUD particularly with regard to the needs of your project.

- You must have telnet access to stay sane now and in the
future, you might need to look at the apache error logs or maybe compile your own perl modules some time. Rackspace is nice, I have not used them but have been very impressed with their responsiveness at night. ssh is better than telnet if you can get it. I recommend linux with RAID 5 disk array.

- Unless you must do clickpath tracking or very high volume
accesses, don't use mod_perl on this project. It is a lot
of fun but you need to be able to restart the server and
it takes a lot longer. There is not a ton of documentation compared to the number of issues that come up with it, so experience is important.

- Don't do credit cards in the beginning. It is complicated
and other companies and laws will be involved.

- As for project schedule, you may have trouble on this project if you allow the client to eat up your development
time with figuring things out. Development time starts after the spec is signed.

- You need to get a dialogue going and confirm your understanding of what they are saying. The client has
to participate in developing a detailed specification,
and must also be coopted to promote success of the project by the following activities:

- Signoff on screen mockups and detailed functional
descriptions before development

- Signoff on delivery of parts, if you can handle that.

- You make weekly risk assessment reports ( a number 1-10
may be good) and use "that increases risk" and "decreases risk" when new functionality or deadlines are
mentioned. The idea is that you do not want to get behind
the eight ball trying to defend yourself against an unrealistic deadline from feature creep or some other problem that may crop up.

- Check out some of the ideas of XP programming philosophy, it has old and new ideas and may give you some
ammunition if you use some of them.

- It will probably take 2-3 times as long as you think. Keep a schedule with expected man-days per task, including testing on your side, bugfixes, testing on their side, content delivery, whatever. Actually content delivery is in addition to that. The variables you generally have include the number of people on the project, the budget, the time until delivery, the quality/robustness of the code, and the SCOPE. Scope means how much you are going to do. It is easiest usually to change scope so you must get
the client involved so they do not make it difficult to change scope.

- There was a very good recommendation above that you look at how your man-days projections match reality and then multiply the difference as a time stretching factor across the entire project. I didn't believe it when I first heard this but it is true.

- Make different phases of the project. The first phase
is consulting which you should get paid for, it involves making the spec and figuring out what they want down to screen mockups (line and text is fine). Also a designer is
probably going to be involved. Make sure of what you are
responsible for, including do you write HTML. Supported
browser types may be another thing to make sure of up front.

- The different phases will make it possible to move functionality to a later phase if you need to. Have the client set priorities (no, they can't say 100% or nothing) so that you can identify what core functionality must be perfect by launch, what is best effort, and what is later phase. Then you can get them to put feature creep functions
into the next phase, or get them to change other variables like schedule or maybe remove a different function.

- Get feedback as soon as possible from other people in
organization so you don't get nailed at the end. Also try to decide as much of the software architecture as early as you can, preferably well before signoff on the spec.

It looks like this is a dangerous project if you don't think of some of the above things. You will have to help your client figure out what he/she wants, and you are only working part time while there is a learning curve for some of the things you will be doing. I would say give yourself a large buffer, start the clock after signoff on the detailed spec, and get the client involved in the process so
they can see what will happen if they become unrealistic about things. Don't handicap yourself in the beginning by
agreeing to a lump payment at the end (especially if you are
meeting milestones and have a consulting report deliverables in the beginning), and if schedule is really important then discuss the possibility of budget changes to add another guy if you really need to at some point.

Have you looked at Positive Internet? Full ssh access, mysql, mod_perl, the works. You can add your own modules, but normally you're better off asking the very helpful tech support guys to add the module to the server so everyone can use it.

From experience on a cgi/database heavy site (http://www.moonfruit.com), mod_perl is *essential* if you are anticipating scaling to larger numbers or users. "use strict" and you'll have no problems with mod_perl.

MySQL has limitations, but is cheap. No transactions (OK - no "normal" transactions...), no stored procedures, no sub-selects (ie no "SELECT * FROM users WHERE user.id IN (SELECT userid FROM admin_users)". But it *is* free, and it *is* included in many web hosting packages

You won't get a decent site done in your spare time in four weeks. There are issues you haven't thought about yet. For example, have you thought of including an admin interface for the people running the site to delete/edit users, get stats, change passwords etc? Of backing up the data and being able to recover it?

There is very little I can say that hasn't already been said as you received a lot of great advice on this. The one thing that I can offer is that when giving project completion dates, is to qualify them. Once you figure out the spec, you can give a date but specify that that will be the date, provided they make no changes. I have found too many times that once a project is underway, the client wants to make a major change in how they want to do business based on something they read or something their kid told them at dinner. Thank god that Perl is so flexible!

instead of starting from scrach, maybe something like what is avaliable at everything development with some
modifications could help you out. remove some features, add some. It might save some time by giving you a start.

Stuffy
That's my story, and I'm sticking to it, unless I'm wrong in which case I will probably change it ;~)

It strikes me that you might have to be a bit of a salesman to your client: Tell them that using some sickly ISP will slow the project. I've hosted sites at (and been reasonably-pleased with) Rackspace since February 2000; having that much control over your environment is invaluable. You'll thank yourself when you're running into problems.

Also, this might be heresy in some circles, but you might want to take a look at Slash. I've never looked closely at the quality of the code (I've managed to change what I've wanted without problem), and it has a bad reputation for clarity, but it is a working system that you might want to borrow from. Remember that it's GPL'd.

I have drafted a proposal, splitting development into phases. I have also requested to use rackspace for hosting. I am currently waiting to hear what the outcome is. I am also looking at purchasing forwardSQL as suggested by an anonymous Monk.

If all goes well I will have enough time to really sit down and play with Perl, instead of frantically trying to debug everything.

Slash is interesting and I will look at it.

I think my biggest problem right now is to decide what exactly to use.
Will I use HTML::Template, Mason etc. ?
will I use mod_perl?
Will I use Mysql? (I still think I will)
If I do get a dedicated server to play with, what other possibilities are out there to do this.

At the moment I know that I am going to use Perl, CGI.pm and DBI with a database (probably MySQL).

Rather than leaving yourself screwed like that, why not deal with a
hosting site that will give you ssh access and allow your own
modules? Try www.mhost.com. You can do pretty much what you
want there, within reason. Plus I personnaly know that all
of the Perl modules you could want are installed - I made
him install em. And they run MySQL.

What does this little button do . .<Click>;
"USER HAS SIGNED OFF FOR THE DAY"

One of the most difficult things to do when contracting is understanding the wants of the client. Most people are not thinking that they need to explicitly detail what they imagine the project to be.
I think they somehow feel that you can magically read their mind.

With this in mind you need to do everything in your power to understand exactly what they want. This means not only getting the "details" in writing but also spending time asking the what if's...

Try to get the thing working in your mind before signing any contracts. Also, most contractors are very comfortable with the language before they dive into a major project. I feel by what you have stated that
there are too many unknowns at this time. It seems like it is diffcult to find ISPs who will let you have access to Perl or a cgi-bin directory these days.

Most contractors have written so much code that they have their own library of subroutine skeletons and modules that they are very comfortable meeting a project deadline. I am not saying that the new programmer will
not be able to do it but the contractor new to the language should allow a forgiving buffer time.

The bottom line is KNOW what your customer wants and understand ALL of your limitations and don't try to pull a rabbit out of your hat. Then it may be a little easier. Good Luck.

It just ringed with me. It is amazing, how often it happens with non-experienced users: When they explain specifications, they explain common cases, but forgot to mention any irregularities, saying these cases are very rare and they did not want to confuse you more. As it happens, these irregular cases might cause complete redesign to meet these "hidden" requirement.

I read somewhere that if God, when creating universe, will ask for sign-on on complete requirements, we will be still waiting to implement.

Author proposed "Genesis approach" (RAD): To create something simple in just 10% of available time, covering maybe only 20% of needed functionality, get feedback from customers, and let it grow. This approach assumes you have ready-made tools and tested procedures ready "to hit the road".

Still, IMHO you may consider this especially in your case, when you are not sure about tools to use. Explain your customer that it might be it may take longer to implement full 100% of what s/he has in his mind, but you cannot read his/her mind, and both you and s/he will have better feeling what needs to be done, what are priorities and what features are not feasible.

First, know that HTML::Mason does NOT need to be run under mod_perl. It's a bit nicer if you can, but it's still wonderful if you have to run under CGI. I'm doing this with www.globalview-intl.com, which is hosted at www.korksoft.com.
Korksoft's hosting plans, by the way, are very reasonably priced, and they offer a lot of great stuff including Mason, MySQL, ssh access, and so on. They're also good about installing Perl modules at your request. Getting support can be slow, but it comes eventually and this drawback is outweighed by the rest.
Viel Gluck,
Dylan Oliver