Harry Truman once said: "It's amazing what you can accomplish if you don't care who gets the credit."
This award recognizes Effective Altruists with $5,000-$10,000 prizes for declining credit in ways that can't be publicized directly, and where their work is more impactful because of that.
Example #1: Sam toils behind the scenes and makes a breakthrough on an important problem. Sam suggests the idea to and then gives credit to a more well-known person/org because that leads to the breakthrough being more widely accepted.
Anyone that knows what happened, including the person/org that gets credit, can nominate Sam for the EA Truman award on <a title="‌" href="http://super-linear.org/">Superlinear</a>. Superlinear passes on the nomination to a committee of well-respected EAs from diverse backgrounds. If one of them verifies that Sam actually did make a breakthrough and allowed someone better placed to take credit to increase impact, Superlinear awards Sam $10,000.
Our committee members are: Eliezer Yudkowsky, Peter Wildeford, Spencer Greenberg, Gregory Lewis, Cate Hall, Ozzie Gooen, Luke Freeman, and Julia Wise.
Example #1: John is an EA who works for the government. There are political or career consequences if it is publicly acknowledged that he’s working on something potentially controversial. John contributes an important innovative idea to AI governance, and helps make it happen behind the scenes. One of the people he worked with nominates him for the EA Truman Award. Then, a committee member like Julia Wise asks two people she trusts, Alice and Bob, to investigate. They confirm John’s contribution. Alice and Bob conclude that this qualifies for the prize, but that John cannot receive prize money due to the government job. Superlinear announces “Julia Wise, Alice, and Bob recommend a non-monetary Truman prize to be awarded for John’s contribution to AI governance.”
If you can’t make an EA forum post about someone receiving credit for doing something noteworthy, this is the prize for you.
Payout mechanism:
<ol>
<li>You see someone that deserves the award, so you nominate them by submitting their info below (you can keep things anonymous, but make sure enough detail is provided that the event is clearly verifiable by someone likely to be known.).&nbsp;</li>
<li>If your submission looks promising, we’ll pass it along to relevant members of our committee of 5-10 well-respected EAs who can verify the event actually happened. So, if the Truman Award nominee is in the biosecurity field, we’d forward it to our biorisk specialist committee member.</li>
<li>The committee member selected chooses 2 additional people (that don’t have to be on the committee) they trust to assist, to allow them to, for example, delegate verification, and discuss the award decision and how much detail to provide publicly. The 3 people involved in the decision would inform Superlinear and sign off on the award. The people involved in the decision would always be public. For example, it might be: "Gregory Lewis, Andrew Snyder Beattie, and Tessa Alexanian award $5,000 to John Doe for an unspecified Biosecurity project which was spearheaded by others," in others it would be "Toby Ord, Hilary Greaves, and Abie Rohrig award a $5,000 Truman Award to an undisclosed recipient, to be named no sooner than 5 years from now, for work which will remain anonymous."</li>
</ol>
Theory of change: Rewarding people for maximizing impact over credit increases the health and future effectiveness of the community.
Example #2:&nbsp;Max has a criminal record and troubled past. He’s reformed now, but his background makes him a liability for any person or org to publicly associate with him. He silently does good work behind the scenes, so someone that knows him nominates him for The Truman Prize on the basis of a specific critical contribution which was made to a now successful larger project. The committee awards the prize, and names Max, likely without naming the specific work done.
Example #3:&nbsp;Steve has extreme political beliefs. It is risky for any person or org to work with him due to reputation risks. Steve knows this, but does apolitical high impact work behind the scenes anyways. Someone that knows Steve nominates him for The Truman Prize on the basis of a specific project which was not previously disclosed. The committee awards the prize and discloses the project, but not the individual, or vice-versa, to avoid undermining the project.
Example #4:&nbsp;Morgan has recurring depression. Therefore, she does not want to work or associate with any specific people or orgs because she doesn’t want to let them down due to an episode. Morgan does a lot of high impact work for free, and gives credit to others who are better placed to continue executing on the project. Someone nominates her for the Truman Prize on the basis of two specific projects that were done without credit. The committee members award the money, and names the individual and the project.

The Truman Prize

Make a substantial contribution to the <a title="‌" href="https://www.lesswrong.com/posts/5bd75cc58225bf0670375575/the-learning-theoretic-ai-alignment-research-agenda">learning-theoretic AI alignment research agenda</a>.
<a title="‌" href="https://alter.org.il/">ALTER</a>'s <a title="‌" href="https://www.alignmentforum.org/users/vanessa-kosoy">Vanessa Kosoy</a> is offering a prize for the best substantial contribution to the<a title="‌" href="https://www.lesswrong.com/posts/5bd75cc58225bf0670375575/the-learning-theoretic-ai-alignment-research-agenda"> learning-theoretic AI alignment research agenda</a>.
Deadline: October 1, 2023.
Depending on the quality of submissions, the winner(s) may be offered a position as a researcher in ALTER (similar to<a title="‌" href="https://www.lesswrong.com/posts/it5odhMKY6xYLrFZD/closed-hiring-a-mathematician-to-work-on-the-learning"> this one</a>), to continue work on the agenda, if they so desire.
Topics The research topics eligible for the prize are:
<ul>
<li>Studying the mathematical properties of the algorithmic information-theoretic<a title="‌" href="https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/shortform?commentId=ovBmi2QFikE6CRWtj"> definition of intelligence</a>.</li>
<li>Building and analyzing formal models of value learning based on the above.</li>
<li>Pursuing any of the<a title="‌" href="https://www.lesswrong.com/posts/gHgs2e2J5azvGFatb/infra-bayesian-physicalism-a-formal-theory-of-naturalized#Future_research_directions"> future research directions</a> listed in the article on infra-Bayesian physicalism.</li>
<li>Studying<a title="‌" href="https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/shortform?commentId=KKiEeGFMisyfrEgqX#6BhuJdyMScbyQeeac"> infra-Bayesian logic</a> in general, and its applications to infra-Bayesian reinforcement learning in particular.</li>
<li>Theoretical study of the behavior of RL agents in<a title="‌" href="https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/shortform?commentId=uZ5xq73xmZSTSZN33"> population games</a>. In particular, understand to what extent<a title="‌" href="https://www.lesswrong.com/posts/zB4f7QqKhBHa5b37a/introduction-to-the-infra-bayesianism-sequence"> infra-Bayesianism</a> helps to avoid the<a title="‌" href="https://www.jstor.org/stable/2951492"> grain-of-truth problem</a>.</li>
<li>Studying<a title="‌" href="https://www.lesswrong.com/posts/5bd75cc58225bf0670375058/superrationality-in-arbitrary-games"> the</a><a title="‌" href="https://www.lesswrong.com/posts/S3W4Xrmp6AL7nxRHd/formalising-decision-theory-is-hard?commentId=3yw2udyFfvnRC8Btr#3yw2udyFfvnRC8Btr"> conjectures</a> relating superrationality to thermodynamic Nash equilibria.</li>
<li>Studying the theoretical properties of the<a title="‌" href="https://www.lesswrong.com/posts/3qXE6fK47JhSfkpnB/do-sufficiently-advanced-agents-use-logic#fEKc88NbDWZavkW9o"> infra-Bayesian Turing reinforcement learning</a> setting.</li>
<li>Developing a theory of reinforcement learning with<a title="‌" href="https://arxiv.org/abs/1907.08461"> traps</a>, i.e. irreversible state transitions. Possible research directions include studying the computational complexity of Bayes-optimality for finite state policies (in order to avoid the<a title="‌" href="https://cstheory.stackexchange.com/questions/41396/is-bayes-optimal-rl-of-a-finite-set-of-dfas-feasible"> NP-hardness</a> for arbitrary policies) and<a title="‌" href="https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId=ndjGcYd39SmYYsHme"> bootstrapping</a> from a safe baseline policy.</li>
</ul>
New topics might be added to this list over the year.
Requirements The format of the submission can be either a LessWrong post/sequence or an arXiv paper.
The submission is allowed to have one or more authors. In the latter case, the authors will be considered for the prize as a team, and if they win, the prize money will be split between them either equally or according to their own internal agreement. For the submission to be eligible, its authors must not include:
<ul>
<li>Anyone employed or supported by ALTER.</li>
<li>Members of the board of directors of ALTER.</li>
<li>Members of the panel of the judges.</li>
<li>First-degree relatives or romantic partners of judges.</li>
</ul>
In order to win, the submission must be a substantial contribution to the mathematical theory of one of the topics above. For this, it must include at least one of:
<ul>
<li>A novel theorem, relevant to the topic, which is difficult to prove.</li>
<li>A novel unexpected mathematical definition, relevant to the topic, with an array of natural properties.</li>
</ul>
Some examples of known results which would be considered substantial at the time:
<ul>
<li>Theorems 1 and 2 in "<a title="‌" href="https://www.lesswrong.com/posts/aAzApjEpdYwAxnsAS/reinforcement-learning-with-imperceptible-rewards">RL with imperceptible rewards</a>".</li>
<li>Definition 1.1 in "<a title="‌" href="https://www.lesswrong.com/posts/gHgs2e2J5azvGFatb/infra-bayesian-physicalism-a-formal-theory-of-naturalized">infra-Bayesian physicalism</a>", with the various theorems proved about it.</li>
<li>Theorem 1 in "<a title="‌" href="https://arxiv.org/abs/1705.04630?context=cs">Forecasting using incomplete models</a>".</li>
<li>Definition 7 in "<a title="‌" href="https://www.lesswrong.com/s/CmrW8fCmSLK7E25sa/p/YAa4qcMyoucRS2Ykr">Basic Inframeasure Theory</a>", with the various theorems proved about it.</li>
</ul>
Evaluation The evaluation will consist of two phases. In the first phase, Vanessa will select 3 finalists. In the second phase, each of the finalists will be evaluated by a panel of judges comprising of:
<ul>
<li><a title="‌" href="https://www.alignmentforum.org/users/adamshimi">Adam Shimi</a></li>
<li><a title="‌" href="https://www.alignmentforum.org/users/diffractor">Alexander Appel</a></li>
<li><a title="‌" href="https://danielfilan.com/">Daniel Filan</a></li>
<li><a title="‌" href="https://www.alignmentforum.org/users/vanessa-kosoy">Vanessa Kosoy</a></li>
</ul>
Each judge will score the submission on a scale of 0 to 4. These scores will be added to produce a total score between 0 and 16. If no submission achieves a score of 12 or more, the main prize will not be awarded. If at least one submission achieves a score of 12 or more, the submission with the highest score will be the winner. In case of a tie, the money will be split between the front runners.
The final winner will be announced publicly, but the scores received by various submissions will not.
Fast Track If the prize is awarded, and at least one author of the winning submission is interested in a researcher position in ALTER, they will be considered for it, although this is not an offer or guarantee of employment. In fact, making such hires to continue to advance the agenda is my foremost reason for organizing this prize.
If multiple winning authors are interested in researcher positions, we will consider hiring at least one of them. It is also quite likely we will consider hiring all of them, but this depends on our financial and organization ability to support that number.
For additional details about the position, see our previous<a title="‌" href="https://www.lesswrong.com/posts/it5odhMKY6xYLrFZD/closed-hiring-a-mathematician-to-work-on-the-learning"> hiring announcement</a>.
Assistance We do not promise to provide any guidance or mentorship to contestants. In fact, identifying researchers that can work with minimal guidance is one of the advantages of this process. However, Vanessa expects to be usually available for providing comments on well-written research proposals. Contestants are encouraged to write such proposals in case of any doubt about the eligibility of their research. Moreover, technical questions pertaining to the learning-theoretic agenda can be asked either on the MIRIx Discord server (where either Alex or Vanessa often answer them), or as comments on the relevant posts by Diffractor (Alex) or myself. Invites to the server are available to good-faith contestants upon request.
If you wish to contact us about either a research proposal or an invite, please write to rot13 of <a href="mailto:inarffn@nygre.bet.vy">inarffn@nygre.bet.vy</a> and attach a CV plus any other relevant background about yourself.
Please indicate your interest in working on this prize on below or comment to find potential collaborators.
Good luck!
Donors can increase the prize pool by adding their contribution commitment below.

The ALTER AI Alignment Prize

Our World in Data (OWID) is one of the world’s largest open online repositories of data, and it's an incredible source of information about both humanity’s progress and challenges. Your forecasts will be used to generate likelihoods of specific outcomes in humanity’s near- and long-term future. And throughout the tournament, a team of <a href="https://www.metaculus.com/help/faq/#what-are-pros">Metaculus Pro Forecasters</a> will also predict alongside the public tournament.
A $20,000 prize pool will reward accurate forecasts on 1 and 3-year outcomes and insightful comments on 10, 30, and 100-year outcomes.

Forecasting Our World in Data

Receive up to $1,000 for writing a short essay that uses the tools of theoretical economics to contribute to the AI alignment problem.
Many think that humanity will likely create an artificial superintelligence within our lifetime.&nbsp; If we align this superintelligence’s values with our own, we can significantly improve the lives of everyone.&nbsp; If, however, we do not succeed in making this superintelligence friendly towards humanity, we have <a title="‌" href="https://www.youtube.com/watch?v=tcdVC4e6EV4">strong</a> <a title="‌" href="https://www.youtube.com/watch?v=hEUO6pjwFOo">theoretical</a> <a title="‌" href="https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf">reasons</a> for thinking it will exterminate us or lock in a bad future for humanity.
This prize hopes to encourage those who know graduate school level microeconomics to think about how to apply theoretical economics to AI alignment.&nbsp; The ideal prize applicant is an economic PhD student or recent economics PhD graduate who has read about the AI alignment problem and thinks something from his or her economic toolkit could contribute to a solution.
Once an applicant has an idea, it should be possible to write a prize-winning essay (minimum 1,000 words) in under four hours as this prize is meant to make it relatively easy for a newcomer to the alignment field to get feedback on his or her potential contribution.&nbsp; Here are links to two <a title="‌" href="https://arxiv.org/abs/2003.00812">academic</a> <a title="‌" href="https://arxiv.org/abs/1906.10536">papers</a> (much longer than what I would expect for an essay for this prize)&nbsp; I have written to give writers insight into the kinds of contributions I think are valuable. Here are some short essays that would be appropriate submissions for this prize if they made their connection <a title="‌" href="https://www.lesswrong.com/posts/aFaKhG86tTrKvtAnT/against-gdp-as-a-metric-for-timelines-and-takeoff-speeds">to</a> <a title="‌" href="http://mason.gmu.edu/~rhanson/dreamautarky.html">economic</a> <a title="‌" href="https://www.overcomingbias.com/2014/07/30855.html">theory</a> <a title="‌" href="https://www.overcomingbias.com/2008/11/billion-dollar.html">more</a> <a title="‌" href="https://www.alignmentforum.org/posts/Sn5NiiD5WBi4dLzaB/agi-will-drastically-increase-economies-of-scale">explicit</a>.
The essay will be judged mostly in terms of its original contribution to alignment.&nbsp; A paper that makes a small, but clear contribution will be considered better than one that makes a vague but broad contribution by, say, attempting to encapsulate the entire situation.&nbsp; Be clear as to what theoretical economic tool(s) you are using.&nbsp; As most of the contributions to alignment come from computer programmers, please make your essay understandable to them by explaining the relevant economic concepts.&nbsp; Clarity will be valued over rhetorical flourish.&nbsp; Ideally, it should be possible to see how expanding the essay could result in an academic paper or actionable proposal.&nbsp; If relevant, include a section that explains how your essay contributes to the literature, but this section can be much briefer than what would be expected in an academic paper and can use Internet links rather than standard bibliographical citations.&nbsp; Fortunately for potential essay writers, few with extensive economic training have written on alignment so many ideas you could come up with are likely to be original.&nbsp; The essay must be in English, and the relevant tool(s) can not be from programming or statistics.&nbsp;&nbsp;
Submit claims down below. I will read the essay, and either reject, accept, or suggest revisions.&nbsp; After I have accepted an essay, I will ask the author to publish the essay on <a title="smartCard-inline" href="http://LessWrong.com">http://LessWrong.com</a> (although this is subject to change).&nbsp; I will wait to see the comments on <a title="smartCard-inline" href="http://LessWrong.com">http://LessWrong.com</a> before awarding a prize amount, which will be between $100 and $1,000.&nbsp; This prize has limited funding so feel free to contact me before you write an essay to make sure that funding is still available.&nbsp; All prize money will be granted by Nonlinear based on my recommendation.
James D. Miller Professor of Economics, Smith College PhD University of Chicago
This prize is funded by <a title="‌" href="https://www.nonlinear.org/">Nonlinear</a>.

The Econ Theory AI Alignment Prize

A few months ago, Thomas Kwa put together&nbsp;<a title="‌" href="https://www.lesswrong.com/posts/voLHQgNncnjjgAPH7/utility-maximization-description-length-minimization?commentId=cPMXSrnnsbX9qLHEB">a statement</a> of a potential “holy grail” of&nbsp;<a title="‌" href="https://www.lesswrong.com/posts/G2Lne2Fi7Qra5Lbuf/selection-theorems-a-program-for-understanding-agents">Selection Theorems</a> and put a prize on it:
<blockquote>
The ultimate goal of John Wentworth’s sequence&nbsp;<a title="‌" href="https://www.alignmentforum.org/s/ogntdnjG6Y9tbLsNS">"Basic Foundations for Agent Models"</a> is to prove a&nbsp;<a title="‌" href="https://www.alignmentforum.org/posts/G2Lne2Fi7Qra5Lbuf/selection-theorems-a-program-for-understanding-agents">selection theorem</a> of the form:
Premise (as stated by John): “a system steers far-away parts of the world into a relatively-small chunk of their state space”Desired conclusion: The system is very likely (probability approaching 1 with increasing model size / optimization power / whatever) consequentialist, in that it has an internal world-model and search process. Note that this is a structural rather than behavioral property.
John has not yet proved such a result and it would be a major advance in the selection theorems agenda. I also find it plausible that someone without specific context could do meaningful work here. As such, I’ll offer a $5000 bounty to anyone who finds a precise theorem statement and beats John to the full proof (or disproof + proof of a well-motivated weaker statement). This bounty will decrease to zero as the sequence is completed and over the next ~12 months. Partial contributions will be rewarded proportionally.
</blockquote>
I’ll call this the Agent-Like Structure Problem for purposes of this post (though I’ll flag that other researchers might prefer a different formulation of the&nbsp;<a title="‌" href="https://www.lesswrong.com/posts/osxNg6yBCJ4ur9hpi/does-agent-like-behavior-imply-agent-like-architecture">agent-like structure problem</a>). For some background on why we might expect a theorem like this, see&nbsp;<a title="‌" href="https://www.lesswrong.com/posts/6mysMAqvo9giHC4iX/what-s-general-purpose-search-and-why-might-we-expect-to-see">What’s General-Purpose Search, And Why Might We Expect To See It In Trained ML Systems?</a>.
Since Thomas’ prize announcement, I’ve had a couple other people ask me about potential prize problems. Such people usually want me to judge submissions for their prize, which is a big ask - good alignment problems are usually not fully formalized, which means judging will inevitably involve explaining to people how their “solution” isn’t really solving the intended problem. That’s a lot of stressful work.
The goal of this post is to explain the Agent-Like Structure Problem (as I see it) in enough detail that I’d be willing-in-principle to judge submissions for a prize on the problem (see final section for terms and conditions). In particular, I’ll talk about various ways in which I expect people will solve the wrong problem.
…

Clarifying the Agent-Like Structure Problem

Making short videos illustrating ways superintelligent AI starts as a chatbot and goes on to destroying our future.
That’s the blocker for people right now, and it’s just pure lack of imagination.
I'm hereby putting up a $2500 bounty for a short video in this genre if I decide it's good enough for me to tweet.
Hoping to fund more than one winner. DM me to discuss.
To clarify, I'm looking for each video to have a specific step-by-step plot. The plot goes from using a chatbot, to at least 80% total future value destruction.
Like a single short Black Mirror episode (though the format can be anything, e.g. could still be a nonfiction style).

Make a short video illustrating ways superintelligent AI starts as a chatbot and goes on to destroying our future

I would really appreciate it if someone could provide a detailed technical argument for believing P(doom|AGI)≤10%.
I’m hereby offering up to $1000 for someone to provide one, on a level that takes into account everything written here (<a href="https://forum.effectivealtruism.org/posts/THogLaytmj3n8oGbD/p-doom-or-agi-is-high-why-the-default-outcome-of-agi-is-doom">https://forum.effectivealtruism.org/posts/THogLaytmj3n8oGbD/p-doom-or-agi-is-high-why-the-default-outcome-of-agi-is-doom</a>), and the accompanying post (<a href="https://forum.effectivealtruism.org/posts/8YXFaM9yHbhiJTPqp/agi-rising-why-we-are-in-a-new-era-of-acute-risk-and">https://forum.effectivealtruism.org/posts/8YXFaM9yHbhiJTPqp/agi-rising-why-we-are-in-a-new-era-of-acute-risk-and</a>). Please also link me to anything already written, and not referenced in any answers here (<a href="https://forum.effectivealtruism.org/posts/idjzaqfGguEAaC34j/if-your-agi-x-risk-estimates-are-low-what-scenarios-make-up">https://forum.effectivealtruism.org/posts/idjzaqfGguEAaC34j/if-your-agi-x-risk-estimates-are-low-what-scenarios-make-up</a>), that you think is relevant. I’m very interested in further steel-manning the case for high P(doom|AGI).
So far none of the comment or answers I've received have done anything to lower my P(doom|AGI). You can see that I've replied to and rebutted all the comments so far. A lot of people's reasons for low P(doom|AGI) seem to rely on wishful thinking (the AI being aligned by default, alignment somehow being solved in time, etc), or perhaps stem from people not wanting to sound alarmist. In particular, people don't seem to be using a security mindset and/or projecting forward the need for alignment techniques to scale to supehuman/superintelligent AI (that will happen after AGI). Alignment needs to be 100% watertight with 0 prompt engineering hacks possible!

Reduce my P(doom|AGI) with technical, mechanistic reasoning saying why it isn't high by default

<a title="‌" href="https://www.amazon.com/Animal-Weapons-Evolution-Douglas-Emlen-ebook/dp/B00JTIOXQ6/ref=tmm_kin_swatch_0?_encoding=UTF8&amp;qid=1659450193&amp;sr=8-1">Animal Weapons</a> studies arms races in the animal kingdom through the lens of evolutionary biology. I'll pay $500 to the best book summary/review of <a title="‌" href="https://www.amazon.com/Animal-Weapons-Evolution-Douglas-Emlen-ebook/dp/B00JTIOXQ6/ref=tmm_kin_swatch_0?_encoding=UTF8&amp;qid=1659450193&amp;sr=8-1">Animal Weapons</a> that translates the lessons learned from biological arms races to potential modern-day arms races.
Payout conditions: After September 15th, I’ll pay $500 to my favorite book review. Conditions: it has to get at least 35ish upvotes on LessWrong or the EA forum and 15 upvotes for the Alignment Forum. <a title="‌" href="https://docs.google.com/document/d/1VA6ihOr3JNRG0xehB56kJF9N9VVrRakPv2VPzgW3Ln8/edit?usp=sharing">Who am I?</a>
Details, resources, and advice: <a title="‌" href="https://www.amazon.com/Animal-Weapons-Evolution-Douglas-Emlen-ebook/dp/B00JTIOXQ6/ref=tmm_kin_swatch_0?_encoding=UTF8&amp;qid=1659450193&amp;sr=8-1">Animal Weapons</a> studies arms races in the animal kingdom through the lens of evolutionary biology. Write a book summary/review of <a title="‌" href="https://www.amazon.com/Animal-Weapons-Evolution-Douglas-Emlen-ebook/dp/B00JTIOXQ6/ref=tmm_kin_swatch_0?_encoding=UTF8&amp;qid=1659450193&amp;sr=8-1">Animal Weapons</a> and try to compare the similarities between biological arms races and technological ones.
Try to translate the lessons you learn into actionable items for the x-risk community. There are many ideas and mental models in this book that I think the x-risk community should know about. The more people are better informed about the general principles that lead to arms races, the more we can be prepared when it actually happens.
Who am I? <a title="‌" href="https://emersonspartz.com/">https://emersonspartz.com</a>
When you post your summary, include a link at the bottom to the bounty and a link at the top of your post to listen to it via the Nonlinear Library.
“Reminder that you can listen to this post and other EA Forum/LessWrong posts on your podcast player with&nbsp;<a title="‌" href="https://pod.link/1587343144">The Nonlinear Library</a>.”

Write a book review for “Animal Weapons” and cross-apply the lessons to x-risk

Submit a write up critiquing, commenting on, or building on claims in <a title="‌" href="https://en.wikipedia.org/wiki/Holden_Karnofsky">Holden Karnofsky</a>’s <a title="‌" href="https://www.cold-takes.com/most-important-century/">Most Important Century</a> series.
What if we’re currently living in the most important century ever for humanity? That’s the bold claim <a title="‌" href="https://en.wikipedia.org/wiki/Holden_Karnofsky">Holden Karnofsky</a>, co-founder of GiveWell and Open Philanthropy, makes in his <a title="‌" href="https://www.cold-takes.com/most-important-century/">Most Important Century series</a>. &nbsp; We are offering $500 to write ups that meaningfully critique, comment on, or build on the Most Important Century series. By incentivizing you to write up your reactions, we hope to stimulate more discussion about these very big-if-true claims. Such discussion is still young and massively neglected by traditional research institutions, so high-quality work you produce for this prize could realistically break new ground. Whatsmore, by demonstrating and training your ability to think clearly about complicated topics, submitting to this prize could help you land future impactful opportunities.&nbsp;
We encourage you to err on the side of submitting to this prize, even if you doubt your ability to contribute. We expect to reward the majority of good-faith attempts. You can also email <a title="‌" href="mailto:drewspartz@nonlinear.org">drewspartz@nonlinear.org</a> with a rough one paragraph sketch and we’ll let you know if you’re heading in the right direction.
What a prize-winning submission could look like:
<ul>
<li>A critique of a claim made in the most important century. For example, identifying an alternative explanation for a phenomenon or commenting on the technical feasibility of predicted developments.&nbsp;<ul>
<li>Example: <a title="‌" href="https://www.dwarkeshpatel.com/p/the-simmer-scenario?utm_campaign=The%20Blog%20Prize%20Digest&amp;utm_medium=email&amp;utm_source=Revue%20newsletter">This Can Go On Pt 1 &amp; 2</a> by Dwarkesh Patel, arguing that, alongside the three key scenarios Holden outlines – Collapse, Stagnation, and Explosion – another scenario is possible: Simmer.</li>
</ul>
</li>
<li>General commentary on the series. What was your overall reaction to the series? What, if anything, did it make you change your mind about, and what are you going to do about it? How does this connect to your own work?</li>
<li>A first-pass at strategizing about the implications of the most important century. How can the world prepare for what might come this century? You can identify your own research questions or begin tackling one of Holden’s <a title="‌" href="https://forum.effectivealtruism.org/posts/zGiD94SHwQ9MwPyfW/important-actionable-research-questions-for-the-most">important, actionable research questions</a>. These questions are probably too big to answer in one submission, but you can still use them for inspiration by suggesting pieces of the answers.<ul>
<li>Example: <a title="‌" href="https://raisingdust.substack.com/p/zombie-universe?sd=pf&amp;utm_campaign=The%20Blog%20Prize%20Digest&amp;utm_medium=email&amp;utm_source=Revue%20newsletter">Zombie Universe</a> by Toby Tremlett lays out the case for how important it is to understand whether the digital people Holden theorizes about are actually conscious.</li>
</ul>
</li>
<li>Extensions to the series. What is the Most Important Century series missing that adds important context to this potentially pivotal time?<ul>
<li>Example: Zard Wright Weissberg’s post <a title="‌" href="https://35milligrams.substack.com/p/re-the-social-science-section-of?r=1ged9i&amp;s=w&amp;utm_campaign=post&amp;utm_medium=web&amp;utm_source=Revue%20newsletter">Re: the Social Science section of Holden Karnofsky’s Most Important Century</a> discussing the relationship between digital people and the future of social science.</li>
</ul>
&nbsp;
</li>
<li>… Any other writing that meaningfully engages with the Most Important Century series!</li>
</ul>
We expect the best posts to have clear <a title="‌" href="https://www.openphilanthropy.org/research/reasoning-transparency/">reasoning transparency</a>. It’s also good practice to use headings and bullet point summaries to increase readability. We’re looking for something roughly blog post length with a minimum of ~1,000 words.
If you’re unsure about whether your write-up ideas meet these criteria, you can email <a title="‌" href="mailto:prizes@nonlinear.org">prizes@super-linear.org</a> sketching out some of the submission ideas you’re considering. We'll aim to provide feedback on your proposed write-ups within a week.&nbsp;
Further prize details:&nbsp;
<ul>
<li>Superlinear judges will award $500 to all submissions that meaningfully engage with the prompt in good-faith. Again, if you’re unsure about whether to submit to this prize, we’d encourage you to apply.&nbsp;</li>
<li>There is no deadline for this prize. So long as this prize is still posted on <a title="‌" href="https://www.super-linear.org/">super-linear.org</a>, we are still excited to receive new submissions.</li>
<li>We maintain the right to change the prize amount or submission criteria.&nbsp;</li>
<li>We encourage multiple submissions, but entrants can only receive the prize once.</li>
</ul>
&nbsp;
How to submit In the Claim Prize form below, share a public link to your write-up (e.g., EA Forum or Google Docs). We'll get back to you on the status of your submission within two weeks.

Most Important Century Writing Prize

Here's my current best guess at how <a title="‌" href="https://www.alignmentforum.org/tag/infra-bayesianism">Infra-Bayes</a> works:
<ul>
<li>We want to get worst-case guarantees for an agent using a Bayesian-like framework.</li>
<li>So, let our agent be a Bayesian which models the environment as containing an adversary which chooses worst-case values for any of the things over which we want worst-case guarantees.</li>
<li>That's just a standard two-player zero-sum game between the agent and the adversary, so we can import all the nice intuitive stuff from game theory.</li>
<li>... but instead of that, we're going to express everything in the unnecessarily-abstract language of measure theory and convex sets, and rederive a bunch of game theory without mentioning that that's what we're doing.</li>
</ul>
This bounty is for someone to write an intuitively-accessible infrabayes explainer in game theoretic language, and explain how the game-theoretic concepts relate to the concepts in existing presentations of infra-bayes. In short: provide a translation.

Explain Infra-Bayes In The Language Of Game Theory

The goal of this prize is to generate ideas and plans for a long-term, positive future. We are going to pretend for this exercise that the technical part of AI Alignment is already solved. Your mission, should you choose to accept it, is to come up with a design for what you want this AI to be doing. 
Relaxations: 
<ul>
<li>The AI will be safe regarding what you're targeting. Example: if you choose to give an AI the sole goal of eliminating cancer, it will actually cure cancer, and not just kill all humans to stop cancers from developing.</li>
<li>You have some degree of influence, politically or otherwise. You can push your idea pretty far, but you’re not all-powerful. If people would normally really hate your idea, or otherwise have no interest at all in it, that’s a problem.</li>
</ul>
Constraints: 
<ul>
<li>The target of alignment you choose has to be something that major political actors (NATO, China, etc.) would not oppose.</li>
<li>It has to be something that would be within the economic interests of AI companies to implement.</li>
<li>It needs to be something that would not provoke massive public backlash.</li>
<li>You can’t turn off or change the fundamental architecture once implemented. (Meaning if you give it a certain target to be optimizing for, you can’t change the target in the future)</li>
<li>There will be no future intelligence (artificial or otherwise) that can ever surpass it in intelligence. It will continue to run until the stars fade and the Universe grows cold.</li>
</ul>
The criteria for selecting a winner will be determined by myself and others at AI Safety Strategy, based on a few factors.
Simplicity over complexity: The simpler and more effective strategies will be preferred over more complex ones. For example, an idea such as Extrapolated Human Volition will lose a lot of points for its added complexity. This is because we don’t know how robust a hypothetical alignment method might be, meaning aligning simpler objectives might be far easier than complex ones.
Multi-party interests: Ideas that are more likely to be backed by multiple interests will rank higher than ones that only please a few. Inversely, ideas that are likely to get opposition from multiple parties will be ranked much lower. All else being equal, an idea that pleases fewer parties but is opposed by almost no one will be ranked higher than an idea idea that pleases more parties but would be opposed by others. If given the choice, favor minimizing opposition over maximizing support.
Modern cultural compatibility: This means that, unfortunately, weirdness will be penalized, all else being equal. This doesn’t mean you should avoid a possible solution just because it’s weird. But, if you can find a strategy that accomplishes relatively the same things, but is less likely to be perceived as weird/crazy/dangerous to the public, favor doing that instead. All else being equal, minimize weirdness. Cultural compatibility in wealthy, industrialized nations will be favored over less developed/powerful nations. This is unfortunately a reality right now, since AI is unlikely to be developed in Papua New Guinea. If you can find a strategy that is culturally compatible with most/all nations, that’s better.
Long-term future: This one will be the most subjective. But I will try to imagine many different minds, with many different value sets, and will favor ideas that more minds with differing values would be at least moderately okay with (all else being equal). I will also consult with many others at AI Safety Support to gauge an idea of how pleasant/appalling a certain future sounds to them.
Uniqueness: Copies of other peoples’ ideas will not be accepted. Building on others’ ideas is allowed (but be sure to credit them). All else being equal, ideas that are more unique and original will be favored over ones that are not.
Read this <a href="https://jaesonbooker.medium.com/the-three-heads-of-the-dragon-99133281c480">blog post</a> to get a better idea of this exercise (if pay-walled, <a href="https://jaesonbooker.substack.com/p/the-three-heads-of-the-dragon?sd=pf">read it here</a>).
This prize is sponsored by Jaeson Booker and the organization AI Safety Strategy.
Current due date is August 1st, 2023. If not enough submissions are received, or if no submission is very promising, the date may be extended.
Submissions can be made in the form of an EA/LW blogpost, or a Google Doc. And simply link to the submission in the comments, or send it to me directly.
AI Safety Strategy: <a href="https://www.ai-safety-strategy.org/">website</a>, <a href="https://discord.gg/e8mAzRBA6y">discord</a> Jaeson Booker: <a href="https://www.linkedin.com/in/jaeson-booker">LinkedIn</a>

Aligned Objectives Prize

Enable Nathan Young to do a number of normal tasks on Metaculus more easily. People want to forecast, but Metaculus is hard to use. With an easier-to-use front end, it needn’t be.
Necessary (worth 80% of the prize pool):
<ul>
<li>Have all the normal filters of Metaculus</li>
<li>Have a feed, where you can scroll down and forecast, or reapprove a previous forecast</li>
<li>Be able to make private binary questions where there is only the following boxes<ul>
<li>Question</li>
<li>Date with a picker, ‘7 days’ ‘30 days’ 1 year</li>
<li>Resolution Criteria</li>
<li>When submitted, the title box gets the first 15 letters of the question, the close date, and resolution date are the date and every other box gets a “.”</li>
</ul>
</li>
</ul>
Stretch goals (worth 20% of the prize pool)
<ul>
<li>Copy the probability editor for making range forecasts, also avoidable from the feed</li>
</ul>
Give it a better name. Suggestions:
<ul>
<li>MetaculusUI</li>
<li>Metaculus Feed</li>
</ul>
Payout conditions / judges:&nbsp;
Nathan Young will adjudicate. If it meets his satisfaction, it will win. To win the stretch goals a site must have the necessary goals too.

Make Basic Tasks in Metaculus Easier

This is to announce a $250 prize for <a title="‌" href="https://www.lesswrong.com/tag/epistemic-spot-check#:~:text=Epistemic%20Spot%20Checks%20is%20a,has%20to%20give%20the%20author.">spotchecking</a> or otherwise indepth reviewing Jacob Cannell's technical claims concerning thermodynamic &amp; physical limits on computations and the claim of biological efficiency of the brain in his post <a title="‌" href="https://www.lesswrong.com/posts/xwBuoE9p8GE7RAuhd/brain-efficiency-much-more-than-you-wanted-to-know">Brain Efficiency: Much More Than You Wanted To Know</a>
…

Checking Jake Cannell's Brain Efficiency

I'll pay $200 for any post on the EA Forum/Alignment Forum/LW that takes ideas important ideas from crypto/web3 and applies them to x-risk.
Payout Conditions The post has to get at least 25 upvotes (15 for Alignment Forum). Payout subject to my discretion. If you don’t get enough upvotes, email your submission to <a title="‌" href="mailto:prizes@nonlinear.org">prizes@nonlinear.org</a>, and if your ideas are interesting enough, I'll still pay out the bounty.
Who am I?
<a title="‌" href="http://emersonspartz.com/">https://emersonspartz.com</a>
Why am I doing this?
6 years ago, Ethereum co-founder Vitalik Buterin wrote an&nbsp;<a title="‌" href="https://medium.com/@VitalikButerin/why-cryptoeconomics-and-x-risk-researchers-should-listen-to-each-other-more-a2db72b3e86b">important but little-known blog post</a> explaining why Cryptoeconomic and X-Risk researchers should listen to each other more.

Write forum post that cross-applies ideas from crypto/web3 to x-risk

Create a calculator for easily making order of magnitude estimates on mobile using Squiggle
Payout conditions / judges:&nbsp;
<ul>
<li>Be able to write out <a title="‌" href="https://www.squiggle-language.com/">squiggle estimates</a> on a phone really easily<ul>
<li>Squiggle is just a coding language so basically I want a really pleasant UI</li>
</ul>
</li>
<li>Be able to do back of the envelop calcuations really easily</li>
<li>Wrap text</li>
<li>Display the graph always with mean 5% 50% 95%</li>
</ul>
Nathan Young will award the prize if he’s able to use it happily.
Resources:
<ul>
<li><a title="‌" href="https://github.com/quantified-uncertainty/squiggle/discussions">Github</a></li>
<li><a title="‌" href="https://forum.effectivealtruism.org/posts/TfPdb2aMKzgWXFvc3/announcing-squiggle-early-access">Forum post explaining Squiggle</a></li>
<li><a title="‌" href="https://forum.effectivealtruism.org/posts/ZrWuy2oAxa6Yh3eAw/usd1-000-squiggle-experimentation-challenge">$1000 Squiggle Challenge hosted by QURI</a></li>
</ul>
Feel free to submit entries to the $1,000 Squiggle challenge prize above as well.

Create a Squiggle Calculator

We will add $5 to your payout for the first bounty that you’ll publish on our platform. But only for bounties published in May 2023!
Please make sure that the bounty recipient has their payment page linked from their profile to streamline the process.