I was just wondering, have you decided on which judging system to use? I myself would still like to see one or another type of voting, possibly approval voting (http://en.wikipedia.org/wiki/Approval_voting) as suggested by one of the members last year. In either case, I think this is a perfect time to discuss this, as the contest has not been launched and will not be for a while.

As opposed to the way it's been some of the earlier years, let's make sure the rules for judging are clear, reasonable, and fair before the competition starts

I never meant to accuse you of anything, I think you did a great job running the contests. However, circumstances and late rules prevented things from being entirely fair. Vague rules on judging criterion prevented the excellent Xero from making the top ten (top three?), instead sending it way down - only because one of the judges couldn't start it. Had rules said that judge should merely be excluded from that game, the results would have been very different. Clear rules and criterion could have helped!

Besides, I see no point in withholding the judging and/or scoring system from us. The big surprise lies in the games, not in the rules. Should you print them out for us now, we might help you find any flaws and settle any disputes in time before the contest starts. I'm not saying they won't be good, or even great; I'm saying that no man is perfect. There might be things you haven't thought of. Give us a change to look at the rules, and things might be better for all parts. Less confusion for us, less final judging headache for you

most of the judging is based on past experience of the last 4 java 4K contests. this being the 5th contest, I'm making it a point to not involve the community into the mechanics of the contest. for reasons why, take a look at threads such as "who needs judges?" etc.

don't misinterpret that, I'm not going to be the almighty 4K contest tyrant - I will always have my ears open.. but there will be no more "so how do you think we should do this?" kind of threads

edit: on another note, you can't blame just the judging system on Xero's failure (there are many more variables than that). you can also blame the judge who *probably* only gave it one shot. as much of a jbanes fan that I am.. you can also blame him! why? obviously not enough testing.. it is well known that java fullscreen on linux is a gamble.. heh, even on windows! providing a backup "windowed" version could have maybe saved him the bad surprise. it sucks that the only game that *really* felt like a genuine arcade game (probably cuz of the fullscreen) scored so low.

also, removing 1 judge because of 1 game is not a flaw in any system - it simply isn't fair. for example... for various reasons, a while ago I generated a result list without the influence of nonnus29 (mostly because he was a particuarly low voter) to see what the results would be. the difference is HUGE and affects ALL games! check out out:

Approval voting needs *many* people who vote... especially lots of those who played all games. While its certainly a good voting system it doesnt really work all that well in this case. With only a handfull of entries it would be a different thing tho.

€: trimmed voting is good btw. Discarding the highest and lowing scores and using the average of the rest or median where the vote in the middle is taken (if its an odd number of judges the two in the middle get averaged).

When I judged, I tried to 1) use a systematic process and 2) went for a bell curve distribution. It never occurred to me that the other judges wouldn't do the same thing. Some judges just awarded 100 points for every game they liked. I, on the other hand felt there should only be a few 'A' games.

But like I said before, judging in this contest is pretty meaningless as well. There really needs to game categories, because there are just too many types of games entered. People who like shooters vote high for them, people who like driving games vote high for them, people who value technical pinache vote for that etc...

So how about 'Best of' categories? Best shmup, best platformer, best gfx, best etc...

we need more diverse judges, yes. but not categories.. the major flaw last year with the judging system is that not all the judges used the same system, although I did say give each game your 1-100 vote which I thought was clear. but apparently you saw it differently -shrug-

that's one of the reasons this year is not going to have a "wanna be a judge? write your name down on this roster" post. an organized judges board will yield better results since they will all use the same system..

Of course I see why you would want to avoid the threads of the earlier years, with something close to flaming happened (I'm blaming nobody). However, I think that was caused primarily because the rules hadn't been set in time. People hadn't gotten the time to think about the effects of them, and when they came the unclear parts could not be fixed without somebody getting an unfair advantage or disadvantage. That's just my point.

I'm not saying we should decide on the rules using another type of voting - I'm saying, reveal the rules quickly. That way, everyone can see how things will be done, and any minor issues can be fixed.

Quote

the major flaw last year with the judging system is that not all the judges used the same system, although I did say give each game your 1-100 vote which I thought was clear. but apparently you saw it differently -shrug-

That's what I meant. Nobody's perfect. The rules last year were good enough, but the criterion for judging was unclear, despite a good effort to make them clear. If a few more members, perhaps even judges only, had been given the chance to agree on criterion, I think we wouldn't have the case were one judge voted in a different way.

Oh, and categories can be good - don't say no just like that, think about it for a while. Personally, I'd say having just a few categories could be a good thing - much like what you're already doing (a players' choice and a judges' choice). These are categories too, you know. Elaboration on why you don't want more categories would be nice, and I expect too agree with you on that matter.

I'm not just saying "no" quickly, we have indeed tried categories before. for one thing, the categories you're talking about would require more than 50 games IMO, to really make a serious difference.

I will reveal the rules when its time to reveal them. just trust me, I'm getting good at this stuff

concentrate more on what entry you're going to make that's going to win the contest, and I'll worry about the things behind the curtain. I am indeed talking to others behind the scenes, particuarly I'm personally pulling together the judges board myself.. so don't worry about it. like I said earlier in this thread, I'm not doing the "wanna be a judge? sign here" post. that was one of the major issues last year [of why the criterion was skewed]

for one thing, the categories you're talking about would require more than 50 games IMO, to really make a serious difference.

I think you might be right about that, and categorizing games can also be hard. On the other hand I think it would be nice to have some awards like best GFX, Sound, innovative idea, and technically advanced. Doom buggy or that real-time ray-traced game 2005(?) finished somewhere in the middle which is OK since gameplay was nowhere near Miners for example. But I still think they should be encouraged (for being mad enough to implement such an advanced ideas into 4K) and should have won an technical achievement award. That would increase the load on the judges a bit, but I think it might be worth it.

Java 4K isn't really big enough for categories. Out the outside it might support 2:

- Best Game- Best Technical Achievement

but then I think that has the equal downside that it encourages people to just write tech demos rather than games, I thought 4K was about how much fun/enjoyment can be packed into 4K, not whether it's possible to write an academic engine with close to zero gamepay in 4K.

I guess it'd be nice to know what the actual overall aim is meant to be - in the past it's been kinda fluid.

Kev

PS. Personally I hate the idea of any sort of player/community vote on games. The result would be entirely to dilute, unlikely to get good feedback, hard to police and runs the risk faddy/populatrity winning the day - rather than some considered judging by people explicitly taking the time to do it. (Admitedly some of the comments last year were a bit week - "ok")

as much of a jbanes fan that I am.. you can also blame him! why? obviously not enough testing.. it is well known that java fullscreen on linux is a gamble..

Now WAAAAAIIIT a minute here. Are you telling me that the judge was running Linux?!?

Because that needs to be disclosed up front. It changes the entire dynamic of the contest, making several features unavailable. It also forces the participants to test across platforms they may not have available. The latter issue is why the contest has historically required the judges to run Windows, as it's the common denominator system that we can expect all contestants to be able to test on. Because of this, fullscreen has also traditionally been allowed by contestants, not just myself.

I have continued (and will continue) to stick with fullscreen mode for reasons I've gone into in the past, and well documented in my game's manual. If the rules are going to change not to allow this, then they need to be stated before the contest starts, not after several people got lower scores because they didn't count on having to test on a machine they don't have.

Quote

heh, even on windows! providing a backup "windowed" version could have maybe saved him the bad surprise.

1. There's no guarantee that the judge would not have had a crash. (Unless he was running Linux.) While fullscreen tends to crash more often, Java occasionally crashes any time the BufferStrategy pipeline is initialized.

2. From the comments, it doesn't sound like the judge ran the game more than once. Why would he have tried the Windowed version if he automatically assumed that the program was broken?

I don't mind taking my lumps (and I said so last year), but a judge's evaluation of being unable to run a program should not be accepted until he's tried some simple diagnostics, like rebooting. In fact, simply trying a second time is usually enough to clear whatever problem occurred. Only after the judge is certain that he can't run it AND that it's the fault of the coder should his score be accepted. While I appreciate that you wished you could have changed the score, Woogley (something that really wasn't feasible), sending it back to him to have him try it again would have far more of a difference.

Similarly, it would have been prudent to reject the JSquares score until it was determined why the judge needed extra components. If the game was outside of the supported versions of Java and components (it wasn't), then it should have been disqualified. If it was a supported version that was accidently triggering Java to take action, then it should have been worked out. Especially since none of the other judges reported issues in either instance. As it so happens, all the judge needed to do was wait for the 1.4 version of Java to download. He could have easily come back to the game later, and waited for the download to complete in the background.

Basically, 100% of the issues with last year's judging could have been prevented by rejecting the judge's submission until the issues were resolved to a satisfactory end rather than accepting the technical issues at face value. Remember, these are game consoles where everything works right the first time. Even commercial games regularly run into technical issues on computers.

*takes deep breath*

Hi everyone! Ready for another fun and exciting contest? Hope I haven't missed too much while I've been away.

I didn't realise the games only had to be playable on Windows - interesting - I feel some nice new hacks coming on

Historically, judging was required to happen on Windows machines as they were the common denominator. (I think it was year 2 we discussed this? I'll see if I can look it up.) Macs were excluded because not many people had them (not to mention 1.4 wasn't available at the time), and Linux had too many oddities to make for a suitable judging platform.

For example, Abuses 4K Shooter in year 2 was only designed to run on Windows. (Feel free to pop in here Anon666/Abuse. ) Because of this decision, Fullscreen was deemed "okay", and many games went on to use it. (Although you do take a HUGE hit for going fullscreen.) I don't mind if this changes, but the rules should be codified on this up front.

I don't mind if this changes, but the rules should be codified on this up front.

Yeah, seems that's where this cafuffle has come from. Need to get a nice clear target platform specification this year. Personal opinion here would be that it's better to require support for all platforms in the now Java glory days where it's actually possible

Kev

PS. I love the fact you can refer back to "year 2" (and year 1 presumably). 4K it's just a contest, it's a legacy man!

Personal opinion here would be that it's better to require support for all platforms in the now Java glory days where it's actually possible

+1 to that. I do all my work on linux, in fact i only have a litle 8G "Wintendo" partition ready for lanparties in wich i only test the games before posting them.

Even if windows is the main ( actually the only ) gaming plataform, we should encourage the use of java for games taking advantage of one of it's most usefull features: it's (almost) plataform independent.

Need to get a nice clear target platform specification this year. Personal opinion here would be that it's better to require support for all platforms in the now Java glory days where it's actually possible

Depends. Is Linux fullscreen support officially operating now?

Actually, this still does create some issues. Like I said, there's plenty of developers who don't have access to a Mac or Linux machine. While Blah did do some judging on a Mac in Year 3 (something which I admonished him for), he also tested on Windows when it wouldn't work on his Mac. I also remember the Planet Lander game not working on the Mac, and the poor developer struggling blindly to make it work when he didn't have access to a machine.

Sooo... it's a tricky subject that probably needs a bit of discussion.

Quote

PS. I love the fact you can refer back to "year 2" (and year 1 presumably). 4K it's just a contest, it's a legacy man!

Fascinating, isn't it? I wonder, has anyone besides me been in every contest since it began? There has to be someone.

Sooo... it's a tricky subject that probably needs a bit of discussion.

I kinda feel thats the benefit of these forums being the hub for all the activity. There are sooooo many people here willing to test stuff out on various platform. I don't have a Mac or a Linux installation but with the help of folks here (and a few other places) I generally get some pretty rigourous testing done.

Quote

Fascinating, isn't it? I wonder, has anyone besides me been in every contest since it began? There has to be someone. Huh

I was around for Year 2, but didn't manage to get my entry done in time. Year 3, was the big one for me

I kinda feel thats the benefit of these forums being the hub for all the activity. There are sooooo many people here willing to test stuff out on various platform. I don't have a Mac or a Linux installation but with the help of folks here (and a few other places) I generally get some pretty rigourous testing done.

please note that the rules are presented clearly, but the adaptations learned from the year do not appear until the next year. since I have been running the contest, there has not ever been a Windows-only rule, and I don't even remember one. (and don't bother referencing one, because I believe you)

regardless, fullscreen is a gamble, even on windows. the situation with Xero is regrettable, but wouldn't ever be completely avoidable (unless you all want to donate a handful of computers of the exact same hardware specs to me for each of the judges). this is one of the gotchas of using a judges' board that is both voluntary and long distant.

(also, I have been around since contest #1. try not to remember my "rock/paper/scissors" game and other creations.. >_>)

That's good to know. So to be clear, a Linux-compatible version wouldn't have helped at the time?

Quote

please note that the rules are presented clearly, but the adaptations learned from the year do not appear until the next year. since I have been running the contest, there has not ever been a Windows-only rule, and I don't even remember one. (and don't bother referencing one, because I believe you)

I've actually been harping on this for a few years now, so how about we see about getting this codified? Do we require all games to run on all three platforms, or can they optionally only run on specific platforms? The former would mean that game developers would have an added burden of testing all systems, and having to work around platform-specific bugs.

The latter would make the lives of developers easier, but it would force the judges to test on one of the systems the developer claims to support. (For example, Xero would have only been testable on Windows and Mac.) There are tradeoffs either way, but the important thing is that we're clear on what is accepted and what isn't.

Quote

regardless, fullscreen is a gamble, even on windows.

Writing *code* is a gamble. Technical issues can and will occur. That doesn't mean that judges shouldn't make a best-effort to solve the issue. While I was a bit upset that Blah was unable to judge several items in year 3, he did make an effort. Here are a few of his comments:

Quote

JM4K - Full screen game that doesn't work even on windows, crashes outPang 4K - broken - supplied instructions fail to work in windows or mac

He included several other comments related to code that wasn't Mac or Linux friendly, but mostly just dinged them a bonus point. As you may remember, Blah's difficulties were what lead to the creation of the executable rule that forbade code that required batch files or command line sequences. This rule was very effective in preventing the same issue from occuring last year.

Now again, we face a similar issue. How do we want to adapt to the problem this year? My take is that the judge should be required to perform *basic* troubleshooting before his score is accepted on the matter. If we don't want to do that, then other options include:

Allowing judges to abstain.

Use Truncated Mean Scoring

Quote

the situation with Xero is regrettable

It is regrettable, but it also isn't the point. You know that I would argue this whether it was my game or not. (Actually, I might argue a lot less if it was just mine. ) I participate in solving these issues every year.

The point is that a problem developed last year in more than one entry. Several suggestions were offered to solve the issue, some without making significant changes to the existing system. Let's do like we do every year: discuss the matter and make a decision about how it should be solved. Our current system is getting really close to excellent. If we can continue to shake out the bugs, hopefully we can get it near to perfect.

Quote

but wouldn't ever be completely avoidable (unless you all want to donate a handful of computers of the exact same hardware specs to me for each of the judges). this is one of the gotchas of using a judges' board that is both voluntary and long distant.

The distance involved is well understood. My own suggestion is not to make the machines uniform, but to:

1. Make a decision on the allowable platforms for judging.2. Make a clear decision on what that means for the contestants.3. Have the contest manager (in this case, you Woogley ) certify that a judge made a best effort to get a game to run before accepting their score. (Remember, once the scores are published, it's too late for a judge to say, "Oh look, I got it to run!")

1 & 2 were not an issue in last year's contest. However, I think they should probably be resolved now, as Java Gaming is now approaching 100% feature compatibility across platforms. Thus we're more likely to experience issues going forward.

3 is the key point as it relates to last year. My suggestion is a simple one that places no extraordinary burden on the judges, contest manager, or developers. If a judge certifies that he's made a best effort, then by all means ding the contestant for it. I just don't think it's a good idea to be accepting scores that are incomplete due to no fault of the developer.

Quote

(also, I have been around since contest #1. try not to remember my "rock/paper/scissors" game and other creations.. >_>)

I trying to remember. Wasn't there a year or two you didn't submit an entry? Or is that my faulty memory?

3 is the key point as it relates to last year. My suggestion is a simple one that places no extraordinary burden on the judges, contest manager, or developers. If a judge certifies that he's made a best effort, then by all means ding the contestant for it. I just don't think it's a good idea to be accepting scores that are incomplete due to no fault of the developer.

If I remember correctly (I can't go back and check because javaunlimited.net is blocked by my company), the issue with my game, JSquares, was that the judge had JRE 1.5 installed. Since my JNLP file specified that the required version was "1.4" and not "1.4+", he had to download 1.4 in order to run the game. In his eyes, this was my application requiring an additional download, and thus a 0 score. Does this count as a fault of the developer, or a failure of the judge to make a best effort? (In my opinion, the JNLP file was not part of the entry, and thus I should have been allowed a chance to fix the JNLP file).

It seems to me that the judges should be required to meet some compatibility minimum - such as having the required target JVM version installed (1.5 was disallowed, he should have had 1.4 installed).

Not trying to whine about my score - I had a blast last year, and i'm looking forward to this year.

but what you're saying is that I should've let you alter the JNLP *after* the March 1 due date, which isn't fair. this is why you should test, and get your friends to test, and their friends to test. surely somebody has a 1.5 JRE (especially if it was posted on JGO??)

judging is taking a large step in another direction this year, with two new systems in place. if you haven't read about it, they are: 1) chosen judges board and 2) people's choice award

even with the chosen judges board we cannot guarantee a "compatibility minimum" per se, the best I can tell you is test test test. we *could* introduce a "standard" OS for judging, but being a java-themed contest (and thus platform-independent)..

but what you're saying is that I should've let you alter the JNLP *after* the March 1 due date, which isn't fair. this is why you should test, and get your friends to test, and their friends to test. surely somebody has a 1.5 JRE (especially if it was posted on JGO??)

His code was tested and found to be compatible with the 1.4 standard for last year's contest. Merely completing the download of the 1.4 JVM (which was all automatic) would have allowed the judge to complete his job. Even on a dial-up, that shouldn't have taken more than a half-hour to an hour.

I don't understand why validation of the scores is such a big deal.

When I handled them back in 2004, I sent the sheets back to the judges a few times to make sure that their scoring was correct, and that there were no anomolies. Changes were made, and each judge signed off on his scores (sometimes overriding concerns I had) before they went public.

Quote

judging is taking a large step in another direction this year, with two new systems in place. if you haven't read about it, they are: 1) chosen judges board and 2) people's choice award

Sounds cool.

Quote

even with the chosen judges board we cannot guarantee a "compatibility minimum" per se

I don't think anyone is asking for one. Merely guidance on what to expect during testing and judging. Remember, everyone is doing this for fun. Throwing them unexpected curve balls is no picnic.

curveballs aren't being thrown by me, they are just happening. who's to blame? who cares. the "infamous" Xero-Zero judge clicked the link, it crashed his system, he gave it a 0. sucks, but that goes back to using fullscreen is a gamble.. not just across OSs, but across hardware.

no matter what judging system we have, something has the chance of blowing up. we're writing some pretty insane code here. the best compatiblity I can give you is use the recommended Java 1.5. that doesn't mean your code won't break in certain situations, though.

curveballs aren't being thrown by me, they are just happening. who's to blame? who cares. the "infamous" Xero-Zero judge clicked the link, it crashed his system, he gave it a 0. sucks, but that goes back to using fullscreen is a gamble.. not just across OSs, but across hardware.

Whatever. Apparently it's eaiser to say, "we'll spread the blame around, then ignore that issues occurred" rather than looking at ways of improving the contest like we've done in previous years. Perhaps I'm an old dinosaur around here. Sorry I showed up.

This is the 4k competition... It is meant to be fun making them... all this bickering about how it is to be scored seems silly...

The way i see it, the competition is to attempt to create a game in 4k, i.e. the competition is your skills vs that 4096 byte barrier! Gaming is such a subjective thing that I would not hesitate to say that there is no "best" game that is produced.

I definetly do not go into the competition to "win" I go into the competition for the challenge and to see what my "best" can produce. If other people like my game then that is nice, but not a large motivator for me.

This is the 4k competition... It is meant to be fun making them... all this bickering about how it is to be scored seems silly...

The way i see it, the competition is to attempt to create a game in 4k, i.e. the competition is your skills vs that 4096 byte barrier! Gaming is such a subjective thing that I would not hesitate to say that there is no "best" game that is produced.

I definetly do not go into the competition to "win" I go into the competition for the challenge and to see what my "best" can produce. If other people like my game then that is nice, but not a large motivator for me.

java-gaming.org is not responsible for the content posted by its members, including references to external websites,
and other references that may or may not have a relation with our primarily
gaming and game production oriented community.
inquiries and complaints can be sent via email to the info‑account of the
company managing the website of java‑gaming.org