]]>https://haddock2009.wordpress.com/2010/11/11/49/feed/0melioriaHaddock-presentationSummer of Code Wrap-Up.https://haddock2009.wordpress.com/2009/08/26/summer-of-code-wrap-up/
https://haddock2009.wordpress.com/2009/08/26/summer-of-code-wrap-up/#respondWed, 26 Aug 2009 03:46:51 +0000http://haddock2009.wordpress.com/?p=32Status summary: Patches are pending to Haddock and GHC [update: they have been included, and released in GHC 6.12.1 and Haddock 2.6.0]. Processing the guts of Haddock comments gets moved out of GHC to Haddock, although recognizing “– |” etc. are still done by GHC.

In fact, at top-level they’re only recognized stand-alone by GHC (data DocDecl: DocCommentNext, DocCommentNamed, etc.), and Haddock gets to match them up with their definitions (Haddock.Interface.Create.collectDocs). Inside function types, data constructors and record-fields, though, they still have to be parsed to a more precise attachment by GHC, which occurs in compiler/parser/Parser.y.pp. But the interiors of the comments can now be parsed by Haddock, at least. Technically, that is Lexing, Parsing and Renaming RdrName->Name. Not to be confused with Haddock’s later renaming Name->DocName that tries to figure out where the documentation links for each rendered thing (like Int or Monad or concatMap) should point to. Renaming RdrName->Name instead starts with something like “concatMap” or “SomethingSauce.Exts.Int16” and figures out what the original defining module is, given the context in which the RdrName appears. For Haddock, this is determined solely by the modules’s GlobalRdrEnv which contains information like “there was an ‘import GHC.Exts as SomethingSauce.Exts hiding (seq)’” and “This module defines a function or value called ‘foomatic’”. Full renaming needs to be more sophisticated and resolve the right-hand-side ‘x’ in ‘f x = x’ correctly, by looking more places than the top-level, but Haddock comments don’t have any local scope like that. At least not currently.

General important concepts in Haddock:

There is a data Interface (Haddock.Types) for each module processed. This is computed by a sequence in Haddock.Interface, most of which is in Haddock.Interface.Create. This data Interface is used to render the HTML docs for the module (Haddock.Backends.Html . Which uses an old locally-kept version of an HTML combinator library). (Or instead/additionally to HTML, it can make Hoogle info.)

This would be simple if every module were self-contained, but it isn’t. Haddock needs to find out about other modules, in order to link to them and re-export things from them. In order to find out about other modules in the group currently being processed (typically a package), Haddock uses a fold over the group’s dependency graph and passes the depended-upon modules’ Interfaces to Haddock.Interface.Create.createInterface. Note that this has consequences for modules that import each other, although I think it might work acceptably/imperfectly in the presence of .hs-boot files. Haddock also loads, in sequence, each of this group of modules using GHC (The GHC-API, -package ghc), so GHC can tell us all about them.

Cross-package, the situation is more complicated. For tedious reasons having to do with space/time efficiency or ease of implementation or nondeterminism or something, we don’t just save all .hs files and Interfaces and stuff to disk. GHC saves “.hi” files to disk for each module, which tell it about all exported information that’s relevant to a compiler, and a bit more. (For example, it doesn’t include doc comments or remember whether a data was declared GADT-style. Probably. There are some weird things it does remember, like whether a constructor was declared infix.) These .hi files are how it can possibly compile the modules that you’re haddocking now. They also let us look up the declarations and types in those .hi files, incidentally (with GHC.lookupName) — though it’s a conversion effort to turn them into HsDecls (see Haddock.Convert, a new module added by my patches). Haddock, likewise, has to record some information about a module in parallel to the .hi file. Haddock.Types.InstalledInterface contains this information for each module — it’s a subset of Interface that mostly contains docs since GHC’s .hi doesn’t save any information about them. (And we’re lazy/stingy so we still depend on GHC for type information, despite its imperfections for our purpose.) When a module is being processed, its Interface is created, and then the InstalledInterface subset is saved to disk. Actually it’s more complicated, because Haddock, unlike GHC, creates a single .haddock interface-file for each *group* of modules it processes (see Haddock.InterfaceFile). Then when you haddock a dependent module, Haddock loads those .haddock files and looks for info in them rather akin to how it would look for information in a locally imported module’s Interface (though it’s always a different though nearby code-path). At least hopefully Haddock loads those .haddock files; it has to be told where they are on the command-line. Cabal will helpfully do so for you, as long as the depended-upon packages have got any installed haddock documentation!

Okay… I think that’s a general overview for now. Questions? Was I confusing or clear?

]]>https://haddock2009.wordpress.com/2009/08/26/summer-of-code-wrap-up/feed/0melioriaNext Project, Move doc-parsing to from GHC to Haddockhttps://haddock2009.wordpress.com/2009/07/21/next-project-move-doc-parsing-to-from-ghc-to-haddock/
https://haddock2009.wordpress.com/2009/07/21/next-project-move-doc-parsing-to-from-ghc-to-haddock/#respondTue, 21 Jul 2009 17:25:24 +0000http://haddock2009.wordpress.com/?p=29I’ve done the first step now! I made a patch that turns the representation of HsDoc in GHC into a FastString rather than a parsed entity, and deleted the parsing code and made it compile. (HaddockModInfo will need to be FastString-ized also.)

(By the way, this means parsing the interiors of the comments. GHC will still be the one to recognize “– |”, “– ^” and so forth, for this phase of the project, and to attach them to the parsed declarations.)

Next comes the presumably harder part: add support in Haddock! (at least the final-product will need to be full of #ifdefs, in order to keep supporting GHC < 6.11, also.)

]]>https://haddock2009.wordpress.com/2009/07/21/next-project-move-doc-parsing-to-from-ghc-to-haddock/feed/0melioriamore cross-package docs detailshttps://haddock2009.wordpress.com/2009/07/20/more-cross-package-docs-details/
https://haddock2009.wordpress.com/2009/07/20/more-cross-package-docs-details/#respondMon, 20 Jul 2009 05:55:57 +0000http://haddock2009.wordpress.com/?p=26I cleaned up most of the loose edges. I still need to design and implement the GADT records syntax (should be easy) ; I was thinking:

So we need to decide what syntax we want, since there’s a tension between displaying the type of the constructor itself (including any class contexts, as GHC just implemented syntax for!), and having a separate line on which to document each field.

Or maybe explicit forall “Bar :: forall a. Ord a => [a] -> Foo a” since it obviously needs to (conceptually) scope over the following fields.

-Isaac

]]>https://haddock2009.wordpress.com/2009/07/20/more-cross-package-docs-details/feed/0melioriaCross-package documentation going well!https://haddock2009.wordpress.com/2009/07/09/cross-package-documentation-going-well/
https://haddock2009.wordpress.com/2009/07/09/cross-package-documentation-going-well/#respondThu, 09 Jul 2009 05:09:16 +0000http://haddock2009.wordpress.com/?p=23A good deal of success! Here is the current status of my loose edges… Plus I’m sure there are some inevitable syntax niceties (choosing when to display in GADT syntax for example), but using this approach, they can’t be perfect, only pretty-good. (The approach is similar to going through Template Haskell except we found out that TH/Convert had some problems, such as producing RdrNames rather than Names, so I wrote the equivalent function to go directly from TyThing to HsDecl Name). So:(tyThingToHsSyn :: TyThing -> LHsDecl Name) is done (except probably for a few corner cases) — it compiles and works! (under 6.11.something and 6.10.3; I just failed to install ghc-paths for 6.8.2[*]) Paste of my working module here:http://hpaste.org/6717

Important remaining corner cases include:
– It seems that TyCon needs to export a new function (tyConParent :: TyCon -> TyConParent), because there’s no way to get that info presently!
– (isFunTyCon tc || isPrimTyCon tc) : how the heck am I supposed to represent these in a TyClDecl?
– When the HsDecls tree contains something of type HsDoc, do I just leave it empty? Will Haddock-code need to / already does fill that in? I guess I’ll find that out once I test my code on more things…? or what?
– it looks like I neglected to implement class ATs yet.

Also, (parseName “Prelude.->”) crashed GHC with “isDataOcc: check me ->”, so I wasn’t able to test it yet … or is it one of those syntaxes that’s so fixed that you’re not even allowed to export or import it, perhaps?

***

Also I succeeded at integrating it into Haddock. Well, almost. For module List, I have the English documentation being displayed where it should be, and I have the signature displayed: all the signatures are “() -> ()”, . I still need to finish threading the Ghc monad through Haddock.Interface.Create so that I can call lookupName at the right times.
– foible: I needed to convert InstalledInterface’s (HsDoc DocName) to ExportDecl’s (HsDoc Name), so I had to essentially write fmap for HsDoc, because it’s lamely not an instance of class Functor. Where should that fmap-code be put? (can it be derived yet?)
– In Haddock.Interface.Create:
-* Under what circumstances is (Map.lookup (nameModule t) instIfaceMap) == Nothing? Is it if we haddock a package for which its dependencies are compiled, but not, themselves, haddocked?
-* I didn’t figure out where to get ExportDecl’s expItemInstances from (but I didn’t really look around for it yet)

okay that’s enough for the day! I need some sleep! And some feedback would be good. Besides that, I do have a plan: Integrate the two pieces I just did; Test on lots more examples than just haskell98-List; fix the things that I obviously neglected; Make the code a bit more presentable. Any big pieces I’m leaving out that we have to do before starting to integrate my code? (well, hoping that there aren’t any more big unexpected things that break along the way, anyway)

-Isaac

]]>https://haddock2009.wordpress.com/2009/07/09/cross-package-documentation-going-well/feed/0melioriacross-package, Plan Ahttps://haddock2009.wordpress.com/2009/06/25/cross-package-plan-a/
https://haddock2009.wordpress.com/2009/06/25/cross-package-plan-a/#commentsThu, 25 Jun 2009 06:04:35 +0000http://haddock2009.wordpress.com/?p=20I think my next step should be the simpler bit: extend .haddock files to contain their docs. (After doing that, I might have a better guess where best to thread in the type-signatures.)

Which means,

#1. That we need to write the info to .haddock files
#2. that we can parse it.
#3. that InstalledInterface can hold it
#4. find out if there’s anything we need to do to make use of the new info in InstalledInterface… well obviously there is

A test case would be a package, that imports and re-exports anything that has a doc-string. I could probably download haskell98 from hackage and use it as a test case.

This is out of order I’m sure: at least writing the info probably means we have to change InstalledInterface first, for example. That’s what plans are for: trying them and refactoring them when they fail!

P.S. I’m doing an awful job at synchronizing my sleep cycle with SimonMar’s… it’s 2 a.m. U.S. East Coast time here, time for me to go to bed (or past time rather

]]>https://haddock2009.wordpress.com/2009/06/25/cross-package-plan-a/feed/2melioriaCross-package documentation, part 1https://haddock2009.wordpress.com/2009/06/24/cross-package-documentation-part-1/
https://haddock2009.wordpress.com/2009/06/24/cross-package-documentation-part-1/#commentsWed, 24 Jun 2009 00:27:17 +0000http://haddock2009.wordpress.com/?p=17It would appear to the casual observer that Haddock works fairly excellently cross-packages. For example, I haddocked the Haddock code, and all the links to the types from the GHC API point to the haddock-docs in my GHC 6.10.3 installation. (Yes, you need the versions of the dependent packages chosen, installed and haddocked, and this doesn’t always work out optimally online. But that’s not the problem that I’m confronting.)

Anyway, the way we get any links to modules in other packages is because we read their generated .haddock files. They’re binary and haddock doesn’t have any obvious way to make them human-readable, but they correspond to Haddock.InterfaceFile.InterfaceFile. Which is, roughly, a list of Haddock.Types.InstalledInterface, which seems to be the interesting bit.

A Module (not to be confused with GhcModule, which is defined by Haddock) is a low-information type defined in GHC that contains the package name and version and the module name.

HaddockModInfo is just the module’s header description, plus the portability:, stability:, maintainer: fields (HaddockModInfo is defined in GHC, oddly enough: must be parse result. defined in ghc:HsSyn to be precise.)

The rest contain a lot of “Name”s, which is a GHC thing that refers unambiguously to the place an identifier originates. Sufficient for making a link, but not sufficient by itself for copying the named identifier’s docs or type. So passing over them for now… there is one interesting thing left.

A DocName contains a Name and also (if any) the module we’d like to link to in which that name is documented. instDocMap :: Map Name (HsDoc DocName). This gives more info on any number of Name identifiers. (HsDoc provides formatting, DocName provides its references .) There is no type information here at all, as far as I can tell, which will clearly need to be remedied somehow (in Interface, roughly a superset of InstalledInterface, types appears in ifaceDeclMap, though I’m not sure if that’s where they’re retrieved from for HTML-doc-printing). But there’s another big question I need to find out: *which* names are documented in any given module’s instDocMap? The type provides no clue, nor does its current (lack of a) doc string, nor ifaceRnDocMap. I could look everywhere in the code that it’s generated, or I could ask David Waern… who will need to tell me if I said anything confused here anyway

]]>https://haddock2009.wordpress.com/2009/06/24/cross-package-documentation-part-1/feed/7melioriaHow To Navigate Your Code:https://haddock2009.wordpress.com/2009/06/23/how-to-navigate-your-code/
https://haddock2009.wordpress.com/2009/06/23/how-to-navigate-your-code/#commentsTue, 23 Jun 2009 21:47:22 +0000http://haddock2009.wordpress.com/?p=13I’ve always wished I could document my own code with Haddock even when it isn’t exported. Today I was exploring the Haddock code-base (which contains a fair number of Haddock-comments) and wanted an easier way to read it.

First I tried `cabal haddock –hyperlink-source`. It only documented Distribution.Haddock because that’s the only exported module of the library! `cabal haddock -v` told me that it was passing `–hide=all the other-modules` to haddock, which I couldn’t reverse with `cabal –haddock-option`. The only way to “fix” that was to hack haddock.cabal so that all modules appear under an `exported-modules:` clause (which is only allowed under `library`, not `executable`, so I had to spuriously merge the executable section into the library section). Yay! Then I had to remove Haddock.Types’s {-# OPTIONS_HADDOCK hide #-} which was causing a bit of trouble.

Adding –haddock-option=–ignore-all-exports includes non-exported functions in the documentation, which can be helpful or unhelpful. I wish it had a mode to include the un-exported stuff *after* all the exported stuff, with a clear separation. As a side note, –ignore-all-exports also “breaks” docs of Distribution.Haddock and any module that re-exports things from other modules, currently…

HsColour is pretty cool too, but it would be even cooler if it hyperlinked all the identifiers in the source, like Haddock does… hmm.

]]>https://haddock2009.wordpress.com/2009/06/23/how-to-navigate-your-code/feed/5melioriaanother boring update, & question :-)https://haddock2009.wordpress.com/2009/05/26/another-boring-update-question/
https://haddock2009.wordpress.com/2009/05/26/another-boring-update-question/#commentsTue, 26 May 2009 02:01:11 +0000http://haddock2009.wordpress.com/?p=11got a cold, decided to take a break for the weekend, and is only beginning to start looking at the Haddock code (in ghc/utils/haddock which is from darcs.haskell.org/haddock2. Apparently it’s managed by “darcs-all” in the ghc tree…). If there’s any higher-level documentation (online? in the doc directories?) maybe I should find and look at it too. Spend a few hours there, and once I do that, maybe it’ll be time to pick a small bug/feature ticket to fix..