I did this as an experiment when I discovered the Harman Direct Positive Paper. Fortunately, they sell 4x5" sized papers, which fit nicely into standard 4x5" film holders. As the name already implies, it's a direct positive paper, meaning that when you process it, it delivers a positive - as opposed to a negative when using standard paper. You won't need any special developer, you can use the standard paper developer.
As far as I can recall the paper's sensitivity is rated at about ISO 3. You will need rather long exposures. Plus, the paper yield a rather strong contrast, which will be difficult to handle but certainly creates an interesting effect for certain scenes.

Anyway, shooting directly on paper is not any more difficult than shooting on film. You will still need to remove the paper from the holder in the dark in order to process it, just like with film. The only difficulty with the aforementioned paper will be handling the contrast / gradiation. Other than that, very straight forward.

Safelight Conditions

Originally Posted by hirokun

You will still need to remove the paper from the holder in the dark in order to process it, just like with film.

I teach kids pinhole photography as a starting point to learning about photography in general. The advantages of using paper instead of film include: (a) it is cheaper than film, (b) paper can be handled safely in safelight so complete darkness is not necessary, and (c) paper used as a film substitute can be processed just like paper is processed for contact prints or enlargements. No special equipment, chemicals or treatment is necessary.

There is a lot of information about using paper instead of film out there. There are even books about it (do a search on Amazon for "paper negatives"). It is very easy to do and yields pleasing results, pretty much indistinguishable from film to the untutored eye.