Wishful 3D file format

Skyhawk Wrote:ok... so we like MD5, my question is:
what modellers does it work with? how do we convert to it? do we have a cocoa loader for it yet?

Uh, why does it have to be cocoa???
If I had a reference on the format I could create a Carbon loader (Library).
I bet some other poeple on idevgames has more that one person that could
create a Carbon loader.
(not just me )

Personally I haven't spent the time researching the MD5 format so for the reason alone currently I'm not backing it. Of course that can change once I'm educated on the format.

What I'd like to do is run this thread for the next week or so and just pool ideas from members here at iDG with whatever ideas/suggestions/whatever they want to throw into the mix. After this period I'd like to assemble the main group interested in this project and start rationalizing all the gathered information and pump out a decision on what specifically should be done that'd best fit all our needs. From there submit the decision to this thread and again more feedback from the rest of the members and go from there...

It might be that there won't be a newly developed format, just adoption of another format like MD5, or possibly a simpler-meta data "side" format that'd work with MD5 to get the desired result. At this point I just don't know.

So I read through a bunch of forum threads over at Doom3World and other stuff and here are a few of my observations so far about md5. It's still fresh out of the factory and the information on it seems a bit flimsy. The whole buzz about md5 seems very chaotic. The format doesn't seem to be usable in a light-weight fashion. What I mean by this is that it appears to be designed to do skeletal animated characters and that's it. But what's more is that mesh vertices are apparently all defined as weights in relation to joints with no actual spatial coordinates. While I really like this, I'm skeptical that it would be easy for beginners to understand. Also, I don't quite understand if one could just define something as simple as a cube in md5 without having to weight the vertices. Still studying, but it doesn't appear on the outside to be designed for that -although, maybe this isn't an issue if you consider your object to have one joint for each vertex and weighted to 1.0.... Hmm... Again, still studying. Rotations are all described as quaternions, which is great for intermediate to advanced devs but again probably not for the beginner. This might not be a problem in practice for simple stuff. I like the fact that they have the animation file separate from the mesh file so that the animation file can be reused with other characters with the same skeleton. The fact that everything is ASCII is very interesting to me. Why wouldn't they have a binary form as well? One last thing, I wasn't able to find any freely available models since it's so new, which sucks because I'd like to get a look at a real file. Anyway, enough rambling. This seems to be a decent light-reading intro to md5: Doom 3 models

I opened this thread because I am creating my own file format anyway, and I thought why not allow others to benefit from it, too. I don't like md5, as it is very specific.

So still, I ask for what features you want. If you prefer to go with md2, 3ds, obj, or whatever, that is fine by me, but if you think that things could be better, well, I'm all ears. I don't want to create yet another dead-end file format.

After spending some more time looking at md5, I don't think it's going to work for what we want here anyway DoG. I'd like to hear others' input on md5 as well to be sure, but it just doesn't look like the right direction to me now. I'm still interested in developing a new format but I'm also open-minded so I think it's wise that we at least *explored* other options before galavanting off into a new unkown territory.

What kind of specific ideas did *you* have in mind. I have some more of my own but after being smacked for offering suggestions a few times I'm feeling like keeping them to myself for now. I suspect others may see it the same way. It seems too easy for just one person to say, "yuck! That sucks!" and have the idea forever shot down than for all the silent others to speak up and say they like it. The best I've seen so far is support for iGame3d's format, which he, himself, seems to have shot down. Oh, and of course the mighty md5, which nobody here actually seems to fully understand.

All I need is a name (and a three letter namespace prefix would be helpful) and a proposed ASCII syntax and I'd start an Xcode project right now. Going back to what PowerMacX suggested, it should be *very* simple for version 1. I would also think that we should stick to an ASCII format first since that's easier to communicate around, and then develop a binary version around it later if desired. It seems to me that working out the actual syntax is the highest priority right now.

[opinion]
I've been keeping quiet on this as I haven't even tackled a 3D game yet. I have looked before and became disheartened when having to deal with the existing 3D formats out there.

My main gripe about existing formats is that they are all tailored specifically to a product. A lot of them are not generic/general enough to be used as a standard for outside of the product. To make things worse, a lot of them enforce a sort of view as to how the object should live in memory or something, flaws of the original implementation and all. And to add insult to injury, as the product evolves so did the format as new crap gets bolted onto the old crap and the format starts to look a little like madusa.

Game formats are not much better than 3D rendering program formats. The reason being is that they develop the format with the limitations/features of the game that they are made for and tailored for that. They aren't generally flexible.

Personnally I'd love to see a very simple and readable file format. It doesn't need to have the kitchen sink in there but a good subset of stuff that will satisfy 80% of the uses. ASCII is a definite must as it simplifies things and a binary format representation should be a direct mapping to the ASCII to keep it simple. One major bonus with an ASCII option is if the format evolves, older programs can ignore what they don't understand. This is easier to do in ASCII than it is in Binary. However I think if the format is crafted in a clever way this may not be at all an issue.

I personnally do not like XML. I'd rather go with a more LISP like representation as you can match braces in a decent text editor and a parser is a lot simpler. I'm assuming the project is intending to provide all the code from parsing and all. If 3rd party libs are involved then XML may have an advantage but I'm sure some will whine about choice of 3rd party lib. XML has the benefit of being nice to view in a browser which I'll give it that but I still don't really like it.

I'd be very happy just to have a super simple format. I.E. 1 object per file. Don't complicate it with being everything. Defining all the information for one piece of object is hard enough. Plus if the library is done properly, the coder using the lib could use it to work inside a collection of 'object's in a file.

I'd go as far as to make a lib that will do it all from parsing and up (what ever up involves) all in portable standard 'C'/'C++'. (Personnally I prefer 'C', allows for easier binding with Python, .NET etc. enz.)

However I've never dealt with all the funky stuff that 3D cards can do now. So I haven't a clue as to what all will be desired/required. OBJ is close to being a 'nice' format but it has some uglies that I couldn't care for.
[/opinion]

BinarySpike Wrote:I've done some looking into .obj and found it WAAAAY to long...

I had a Soccer ball (each side was about 20 units in width)
that had about 300 lines of code (including returns)

For coding that into OpenGL thats about... 50 lines (including texturing)
...
writing a OpenGL converter would be MUCH easyer that .obj
(Only there would be a compiler so that the thing wasn't in text)

I am not quite sure how to respond to that. I am disturbed. Or isn't it me, after all?

There are trade offs with all formats. The things I like about wavefront .obj is that it keeps vertices, texture vertices and normals unique from each other. In your case texture vertices and vertices are tied closely with the face. In .obj a face has indicies into the vertex, texture and normal lists. This doesn't lend itself well to fast game rendering but does make for a flexible and tighter format when face counts get large.

What I don't like is that if the .obj has multiple unique objects, the way it indexes can potentialy make it a real pain to separate the points, textures and normals of one object from the other. Also I find it confusing when they start changing the material and all. It just seems to me that it had a nice start but evolved badly.

Also it can handle nurbs and splines and the like which adds more complexity to the format. Oh the pain...

* The texture path shouldn't be absolute!!!!! Otherwise the game will only run on players with the same username as the developer >Relative paths<
* Why do you need an "$endface" when $Face already indicates the number of vertex?
* Why use such long "tag" names?
* Why use @s and $s?
* ASCII format is interesting, but it's unrealistic to expect writing your 3D models by hand in a text editor, so it doesn't need to be so "verbose" (ie. in obj you use "v" for vertex, "t" for texture, and so on)

list of key frames ("number of key frames" key frames, where each keyframe contains a
* list of vertex ("number of vertex" vectors)
* list of normals ("number of normals" vectors)(can be void, if the num. of normals is 0 - flat shaded models)

I made an OBJ importer, that can take a series of OBJ and convert it to this format, making each OBJ a key frame.
This is very basic (the code for loading it is less than 60 lines, 40 for displaying an interpolated key frame), but can be expanded.

Zekaric Wrote:The things I like about wavefront .obj is that it keeps vertices, texture vertices and normals unique from each other.

...

It just seems to me that it had a nice start but evolved badly.

I agree.

For example, it makes more sense to me to have a single tag saying how many vertices to follow instead of prefixing every line with a "v" for vertex. Two reasons: 1) I've said this before and I'll say it again, it is often easier from a parsing standpoint to have a single tag coded to say that "there will be eight vertices following this tag" so that memory can be allocated right then and there without having to calculate from some other dependent variable. Or even worse, having to actually parse the file once extra just to count! 2) for something like 200 vertices, having a tag like # Vertex 200 takes up only twelve 12 bytes on disk whereas prefixing every vertex line with a v would take up 200 bytes on disk, plus extra work to lex over the v. It just seems unnecessary to me to have v, or whatever, on every line when I already know darn well what it is I'm parsing.

That reminds me. Wouldn't it be beneficial for file size to *not* use 1.00000 to represent 1.0? Why not just stop listing zeros when the first least significant zero digit is encountered? You wouldn't say 00034.34 would you? I often see something like 52.23200000 in an ASCII file. That should just be 52.232 IMHO. I suppose the argument could be made that it might make the alignment easier on the eyes, but... Does it really? I hardly ever see spacing in the file to accomodate a "-" sign and that screws alignment up anyway. That's not to say that extra zeros shouldn't be allowed, but I would say that it should be encouraged to leave them out.

Here are a couple of ideas for a vertex tag with one example vertex line following each:

Code:

# Vertex 128
1.0, 2.23, 0.0

<Vertex 200>
1.0, 2.23, 0.0

V 320
1.0, 2.23, 0.0

Vertex 150
1.0, 2.23, 0.0

The third and fourth examples would make the parser assume that a non-numeric character denotes the beginning of a tag. Using something like a # or < makes parsing much more efficient however since it is immediately obvious.

PowerMacX Wrote:* The texture path shouldn't be absolute!!!!! Otherwise the game will only run on players with the same username as the developer >Relative paths<
* Why do you need an "$endface" when $Face already indicates the number of vertex?
* Why use such long "tag" names?
* Why use @s and $s?
* ASCII format is interesting, but it's unrealistic to expect writing your 3D models by hand in a text editor, so it doesn't need to be so "verbose" (ie. in obj you use "v" for vertex, "t" for texture, and so on)

I think, just guessing, that the $endface is needed because he could have more than one face present between $face and $endface. The count beside the $face indicates quads or tris follow.

As to verbose nature, that is pretty much a preference I would guess. More for readability than efficiency. Also if you make things too terse you may end up with something that looks like bad PERL code.

ASCII is really meant for eye balling correctness. Easier to do than a binary file.