In this case, the equiv­a­lent is a “car safety” non­profit that goes around to all the car com­pa­nies to help them make safe cars. The AI safety ini­ti­a­tives would at­tempt to make sure that they can help or ad­vise what­ever groups do make an AGI. How­ever, know­ing how to ad­vise those com­pa­nies does re­quire mak­ing a few cars in­ter­nally for ex­per­i­men­ta­tion.

I be­lieve that OpenAI ba­si­cally pub­li­cally stated that they are will­ing to work with any groups close to AGI, but for­got where they men­tioned this.

That makes sense to me. When I said “harder to scale”, I mean harder to “put a bunch on top of each other”. In some ways it’s not as el­e­gant.

Agreed that Im­pact Prizes are one ways that Cer­tifi­cates of Im­pact could work long-term. Like, one group places $100k of Im­pact Prizes for 2030, where it will only be used to pur­chase Cer­tifi­cates of Im­pact.

Hon­estly, the tech­ni­cal in­fras­truc­ture for Cer­tifi­cates of Im­pact would be very similar to that for Im­pact Prizes as I dis­cuss them above. I think both would be re­ally in­ter­est­ing to test at larger scales.

Im­pact Prizes may need less hype though, but may be more difficult to scale.

I imag­ine it would se­lect in part for sales & per­sua­sion, but not more than for other prizes (where you need to do the same for the judges). The mid­dle­men would fo­cus on the fi­nan­cial mo­tive, so I’d ex­pect them to be rel­a­tively sane.

I would re­ally de­sire the eval­u­a­tions and pre­dic­tions/​es­ti­ma­tions to be re­ally good, in or­der to make sure peo­ple fo­cus on the right things.

Gen­er­ally, I feel like there are ac­tu­ally pretty few reg­u­lar en­g­ineer­ing po­si­tions around for EAs (Maybe 8-15), and these both have fairly high bars and re­quire work in the US/​UK.

Small orgs have differ­ent needs to large ones, and most of the EA groups are small. This in part means they want se­nior and/​or en­trepreneurial types.

I do sug­gest that pro­gram­mers learn ML or in­tensely learn Func­tional pro­gram­ming, though not that many available peo­ple seem in­ter­ested in ei­ther (es­pe­cially those who are do­ing E2G out­side of EA jobs.) Either would be a sig­nifi­cant challenge, for one thing.

I’ve worked around this space (cofounded .im­pact), and cur­rently I do recom­mend Michal’s work for this is­sue.

That said, I think it’s less over­looked than it would ap­pear. Vol­un­teers, even tech vol­un­teers are gen­er­ally re­ally difficult to work with. The re­ally good ones tend get to get hired by groups rather quickly, and most of the rest are quite flaky. (Though this may be less true of Eng, which is a bit less in de­mand than other roles now.)

There’s gen­er­ally a lot of over­head for man­ag­ing a tech pro­ject, and do­ing it with some­one who has a good chance of flak­ing out quickly is not that great.

My im­pres­sion is that while there are a bunch of EAs in tech, very few are will­ing to sus­tain a 10hr/​week plus time com­mit­ment; es­pe­cially ones who don’t have a lot of other ex­pe­rience do­ing side pro­jects of that type.

1. I’ve been do­ing a de­cent amount of think­ing & ex­per­i­men­ta­tion in similar work re­cently. I’m per­son­ally op­ti­mistic about non-mar­ket ap­pli­ca­tions like GJP and Me­tac­u­lus. I think that the path for similar groups to pay fore­cast­ers is much more straight­for­ward than similar in pre­dic­tion mar­kets. I think there could be a lot more good work in this area.

2. GJP charges sev­eral thou­sand per ques­tion, but Me­tac­u­lus is free, as­sum­ing they ac­cept your ques­tions. I think the an­swer to this is very com­pli­cated; there are many vari­ables at play. That said, I think that with a pow­er­ful sys­tem, $50k-500k per year in pre­dic­tions could get a pretty sig­nifi­cant in­for­ma­tional re­turn.

3. This is also a very vague ques­tion, it’s not ob­vi­ous what met­rics to use to best an­swer it. That said, if a good pre­dic­tion sys­tem is made, it could help an­swer this ques­tion in spe­cific quan­ti­ta­tive ways. It seems to me that a ro­bust pre­dic­tion sys­tem should be roughly at least as ac­cu­rate as a non-pre­dic­tive sys­tem with the same peo­ple. Long-term pre­dic­tions are tricky, but I think we could have some ba­sic es­ti­mates of bias.

4. This is also a huge ques­tion. I think there’s a lot of ex­per­i­men­ta­tion yet to be done here on many differ­ent kinds of ques­tions. If we could have meta-pre­dic­tions on things like, “How im­por­tant will we have found this pre­dictable item was to have in the sys­tem”, then we may be able to use the sys­tem to an­swer and op­ti­mize here.

5. I’m not very op­ti­mistic about pre­dic­tion mar­kets. This is of course some­thing that would be nice to for­mally pre­dict in the next 1-3 years.

I think one as­sump­tion is that com­pared to the main pres­ti­gious EA po­si­tions now, most jobs are or­ders of mag­ni­tude lower-im­pact per unit time. OpenPhil has spent a lot of time ex­plor­ing op­tions and only found a few pos­si­ble ar­eas, and even some of those (prison re­form) don’t seem as good as AI safety, from what I can tell, in many ways. Un­less there’s some clever EA anal­y­sis that a field is re­ally sur­pris­ingly good, I think the bur­den of proof is on that field to show some sur­pris­ing in­sight; in this case, ed­u­ca­tion. If you have a se­nior role you may be able to do 5x as much, or 15x, but I think the think­ing is that the choice of in­dus­try could make a 50-200x differ­ence.

Quick 2c: I think it’s typ­i­cally as­sumed among many promi­nent EAs that global poverty /​ an­i­mal is­sues /​ long-term is­sues are all a lot more effi­cient than U.S. ed­u­ca­tional is­sues. As such, I’d per­son­ally ex­pect that the main benefits of you do­ing that work, as­sum­ing you will later work in one of the three ar­eas I men­tioned (or meta-work), to come from the first two things you men­tioned (learn­ing & ca­reer cap­i­tal.)

I think it’s in­cred­ibly difficult to have much coun­ter­fac­tual im­pact in the for-profit world. You’re right to have con­sid­er­able epistemic un­cer­tainty.

I’m per­son­ally more in­ter­ested in US struc­tures, mainly be­cause I have a lot more fa­mil­iar­ity with them and ex­pect to spend most of my ca­reer in the US. That said, this post was meant to col­lect gen­eral ad­vice for oth­ers also do­ing similar things, so any thoughts are ap­pre­ci­ated.

Good to see! I used screen flow to record my­self go­ing through the site for the first time, and recorded my re­ac­tions and thoughts. I’m hes­i­tant to post it pub­li­cly (though in­vite you to ever do so, if you want), but sent it in Slack. In gen­eral I en­courage peo­ple to re­view new pro­jects in that way and similar; text feed­back could take more time and be less in­for­ma­tion-dense (un­less you spend a while sum­ma­riz­ing it).

This is re­ally neat, I’m a big fan of the com­pre­hen­sive ap­proach and the doc­u­men­ta­tion style. Will spend more time later look­ing into the de­tails; I’m not an ex­pert in the field and can’t com­ment on the spe­cific meth­ods, but the high-level work seems very rea­son­able.

Side note that it seems kind of dis­mal that wild chimps are ap­par­ently rated with higher welfare than av­er­age hu­mans in In­dia, though I guess the chimp lives may ac­tu­ally be pretty nice, es­pe­cially be­cause there aren’t many of them. On that note, are there other an­i­mal species you think are par­tic­u­larly happy, but didn’t in­clude in this re­port?