Impact Markets

We run this platform

We offer internships and grants to a select few of the best students and graduates from top French universities to work on AI safety for 3 to 6 months. The goal is to upskill them so they can do useful research, attend a program such as SERI MATS at Stanford, or join an AI safety org. We expect that ~4 of them will work on technical research, and ~2 on governance. France has among the best researchers in the world (especially in math and CS) but has no AI safety pipeline yet. EffiSciences is the only org focused on building the AI safety field here, aiming to leverage this untapped pool of strong researchers.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




EffiSciences

We are in the process of evaluating the most impactful area for research and policy development. [On this page](https://www.balsaresearch.com/research-areas), you can see examples of policies that make our shortlist.

## [Repealing the Jones Act](https://www.balsaresearch.com/research-areas/jones-act "‌")

The Merchant Marine Act of 1920 (Jones Act) has crippled our merchant marine and domestic shipping capabilities to only narrow private benefit. Repeal even of manufacturing requirements alone would be transformative, starting with immediately switching traditional coastal trucking routes to cheaper, more eco-friendly and cost-efficient shipping routes, significantly reducing traffic and carbon emissions while lowering costs for domestically produced goods, improving American competitiveness and providing a noticeable bonus to economic capacity and growth. Benefits are so large that private losers could be made whole.

## [Reforming or Re-imagining NEPA](https://www.balsaresearch.com/research-areas/nepa "‌")

The National Environmental Policy Act has noble goals but its core principle of enforcement via lawsuits and veto points leads to continuously expanding needless requirements, costs and delays, including to green projects. What we have now is not what anyone wanted or intended. The underlying structure frustrates most attempts at incremental reform.

Moving towards a time-limited cost-benefit approach with negotiation and collaboration among real stakeholders and a robust independent analysis funded by development interests, followed by a deliberative governmental go or no-go decision, could better safeguard our environment and communities.

## [Rethinking Housing Policy](https://www.balsaresearch.com/research-areas/housing-policy "‌")

Our failure to build sufficient housing in places people want to live due to expanding regulatory burdens, overly restrictive zoning and local veto points (‘NIMBY’) is driving up the cost of living, preventing people from living where they can be most productive and holding our economy back. Making our most productive cities bigger and more affordable would supercharge our economy and prospects for workers. Federal efforts to address this collective action problem have been neglected due to lack of perceived opportunity, but there are more affordances available than people realize, from building codes to conditional subsidies to the rules for mortgage influence and more.

## [Reasonable AI Policy](https://www.balsaresearch.com/research-areas/ai-policy "‌")

AI will transform our world. Along with great promise, it also poses a real risk of human extinction. We are exploring how best to guard against catastrophic outcomes by preventing the training and deployment of dangerously capable systems until we know how to align them with human values and ensure control of the future is not lost. This requires a focus on possession, collection and usage of high end AI-focused computer chips and data centers, as the only practical way to prevent the unsafe training of dangerous systems. Done correctly this can avoid stifling our ability to enjoy the gains from innovation, and also ensure our privacy remains respected. The wrong regulations, poorly targeted, could easily either cripple the existing industry or be captured by it to protect insiders, without protecting us from extinction risks.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Balsa Research

The AI Objectives Institute is a nonprofit research lab of leading builders and researchers. We create tools and projects to ensure that AI and economic systems of the future are built with genuine human objectives. Our aim is to avoid large-scale economic, institutional and environmental catastrophes from misaligned systems while improving human flourishing. 

Founded by the late internet security pioneer Dr Peter Eckersley, the organization collaborates with and employs working groups of academics, scientists, economists, entrepreneurs, and government and civil society advisors. The results are open-source prototypes, products, and research – all of which explore solutions to keep human objectives central to how AI impacts society - at individual, collective and systemic levels.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]

AI Objectives Institute

A policy research organization within Georgetown University’s Walsh School of Foreign Service, CSET provides decision-makers with data-driven analysis on the security implications of emerging technologies. CSET is currently focusing on the effects of progress in artificial intelligence (AI), advanced computing and biotechnology. We seek to prepare a new generation of decision-makers to address the challenges and opportunities of emerging technologies.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Center for Security and Emerging Technology

Epoch is a research group trying to forecast the development of transformative Artificial Intelligence. They try to understand how progress in AI happens and what economic impacts we might see from advanced AI.

They want to enable better governance during this economic transition by informing about the timing of new developments and what levers can be used to influence AI progress.

## Research agenda

Their research agenda includes both empirical and theoretical work.

![projecting-compute-trends.jpeg](https://epochai.org/assets/images/posts/2022/projecting-compute-trends.jpeg)

## Modelling the future of AI

They are developing statistical and economic models to answer crucial questions about the direction and impact of AI in the coming decades.

[LEARN MORE](https://epochai.org/research/forecasting.html "‌")

![compute-trends.png](https://epochai.org/assets/images/posts/2022/compute-trends.png)

## Studying trends in Machine Learning

Curating and analyzing information about significant ML systems like compute budgets, training dataset sizes, cost and performance.

[LEARN MORE](https://epochai.org/research/ml-trends.html "‌")

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]


Epoch

The Future Society (TFS) is an independent 501(c)(3) nonprofit organization based in the US and Europe. Our mission is to **align artificial intelligence through better governance** to ensure that AI systems are safe and adhere to fundamental human values. We develop, advocate for, and facilitate the implementation of AI governance mechanisms, ranging from laws and regulations to voluntary frameworks such as global principles, norms, standards, and corporate policies.

[Our work](https://thefuturesociety.org/our-work/ "‌") spans a range of governance themes, including [AI and the Rule of Law](https://thefuturesociety.org/theme/ai-and-rule-of-law/ "‌"), [European AI Governance](https://thefuturesociety.org/theme/european-ai-governance/ "‌"), and [Global & Corporate AI Governance](https://thefuturesociety.org/theme/global-corporate-ai-governance/ "‌"). Through our work, we collaborate with intergovernmental organizations, governments, companies, academic institutions, and other civil society organizations.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




The Future Society

The Median Group, formed in early 2018, is a 501(c)(3) non-profit dedicated to research on global catastrophic risks. Our focus is on technological and geopolitical changes over the next century, and our aim is to model what such changes may look like and how their most detrimental effects might be mitigated or averted. Current research efforts include models of past and future progress in AI, presented at the [Center for Human-Compatible AI](http://humancompatible.ai/ "‌"), and of the impact climate change will have on global social and political dynamics.

See our [Research](http://mediangroup.org/research "‌") section for more details on specific projects, or follow our [RSS](http://mediangroup.org/feed/rss.xml "‌")/[ATOM](http://mediangroup.org/feed/atom.xml "‌") feed to be notified of work as it’s published.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Median Group

The Global Priorities Institute exists to develop and promote rigorous, scientific approaches to the question of how appropriately motivated actors can do good more effectively, with a particular focus on areas that are not already well addressed by existing academic research.

This line of thought motivates the following vision and mission:

Our Vision: A world in which global priorities are set by using evidence and reason to determine what will do the most good.

Our Mission: To conduct and promote world-class, foundational academic research on how most effectively to do good.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Global Priorities Institute

CAIS exists to ensure the safe development and deployment of AI

AI risk has emerged as a global priority, ranking alongside pandemics and nuclear war. Despite its importance, AI safety remains remarkably neglected, outpaced by the rapid rate of AI development. Currently, society is ill-prepared to manage the risks from AI. CAIS exists to equip policymakers, business leaders, and the broader world with the understanding and tools necessary to manage AI risk.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Center for AI Safety

The Swiss Existential Risk Initiative (CHERI) supports bachelors, masters and PhD students looking to have a positive impact with their career in global catastrophic risk mitigation.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Swiss Existential Risk Initiative

We are a dynamic community of researchers and policymakers from across Europe, united in our efforts to advance AI safety. Our network brings together experts from over 13 countries and counting, all working towards a common goal: ensuring that artificial intelligence systems are safe, ethical, and positively impactful for humanity. ENAIS (pronounced e-nice) distinguishes itself from related organizations by its focus on existential risks from AI.

At ENAIS, we believe that as AI technology continues to advance, it's important to consider the potential risks and challenges that come with it. That's why we are dedicated to researching and promoting best practices in AI safety, ensuring that advanced AI systems are developed and deployed responsibly.

Join us in our mission to make AI a force for good. Together, we can shape the future of this exciting and rapidly evolving field.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




European Network for AI Safety (ENAIS)

The Theory team is led by Paul Christiano, furthering the research initially laid out in our report on Eliciting Latent Knowledge. At a high level, we’re trying to figure out how to train ML systems to answer questions by straightforwardly “translating” their beliefs into natural language rather than by reasoning about what a human wants to hear. We expect to use funds donated via this fundraiser towards supporting the Theory project.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Alignment Research Center: Theory Project

The Evaluations project is a new team led by Beth Barnes, working to build capability evaluations (and in the future, alignment evaluations) for advanced ML models. You can learn more at evals.alignment.org

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Alignment Research Center: Evaluations Project (ARC Evals)

The Forethought Foundation is especially interested in the idea that the primary determinant of the value of our actions today is how those actions influence the very long-run future. It believes that by making the right decisions today, humanity has the opportunity to positively steer civilisation’s trajectory for thousands of years to come.

To this end, the Forethought Foundation is interested in [supporting excellent research](https://www.forethought.org/research-overview "‌") that:

- Defends or criticises the idea that we should primarily care about the very long-run impact of our actions.
- Works out the implications of a long-term view for our decisions today.
- Is of importance for the long-run future of civilisation, even if the research is not directly about the idea of long-term impact.

It supports such research through scholarships and fellowships to students in global priorities research, as well as research grants and prizes for established scholars.

The Forethought Foundation is a project of [Effective Ventures](https://ev.org/ "‌") (as is Giving What We Can — see our [transparency page](https://www.givingwhatwecan.org/transparency "‌")), and works in close collaboration with the Global Priorities Institute at Oxford University.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Forethought Foundation for Global Priorities Research

Longview Philanthropy designs and executes bespoke giving strategies for major donors. Its grant recommendations are driven by the conviction that we can use evidence and reason to find the highest-impact opportunities in the world.

Everything Longview Philanthropy offers is free, independent, and reviewed by external experts, and is guided by its [core values](https://www.longview.org/about "‌"):

- Longtermism: Our collective future could be extraordinarily good or inordinately bad. Longview believes this generation has the power to influence which path humanity takes, and it is essential that we act responsibly.
- A scientific mindset: Wherever possible, Longview is informed by the latest research from the natural and social sciences, and it acts on the best available balance of evidence.
- Intellectual honesty: Informed by world-class expertise, Longview only makes the grant recommendations that it thinks are best. It does not merely tell its philanthropists what they want to hear.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Longview Philanthropy

The Simon Institute for Longterm Governance (SI) is a nonprofit based in one of the few hubs for global governance: Geneva, Switzerland. While most think tanks focus on supplying information, SI specialises in supporting policy networks in processing information overload, with a focus on the world’s most pressing problems.

Specifically, SI:

- Supports international civil servants and diplomats with training programmes that both facilitate group decision-making by practising key skills and tools, and foster an understanding of extreme risks and the significance of their mitigation. (See SI's [theory of change.](https://www.simoninstitute.ch/blog/post/our-theory-of-change/ "‌"))
- Supports longtermist actors in their engagement with [international policy processes](https://www.simoninstitute.ch/blog/post/setting-expectations-for-extreme-risk-mitigation-through-policy-change/ "‌") through research, training, and networking.
- [Analyses international policy processes](https://www.simoninstitute.ch/blog/post/setting-expectations-for-extreme-risk-mitigation-through-policy-change/ "‌") and how to improve their consideration of future generations.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Simon Institute for Longterm Governance

On this podcast, I ([Daniel Filan](https://danielfilan.com/ "‌")) have conversations with researchers about their papers. We discuss the paper and hopefully get a sense of why it’s been written and how it might reduce the risk of artificial intelligence causing an [existential catastrophe](https://en.wikipedia.org/wiki/Global_catastrophic_risk "‌"): that is, permanently and drastically curtailing humanity’s future potential.

This podcast launched in December 2020. [Finan Adamson](https://www.finanadamsonvoice.com/ "‌") edited the podcast between February 2021 and February 2022, and remains the audio consultant. Starting in March 2022, editing is done by Jack Garrett. As of August 2022, Amber Dawn Ace helps with transcription.

You can subscribe to AXRP by searching for it in your favourite podcast provider. To receive transcripts, you can subscribe to this website’s [RSS feed](https://axrp.net/feed.xml "‌"). You can also follow AXRP on twitter at [\@AXRPodcast](https://twitter.com/AXRPodcast "‌").

If you like AXRP, you might like its sister podcast, [The Filan Cabinet](https://thefilancabinet.com/ "‌"), where I interview people about a wide range of topics I’m interested in.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




AI X-risk Research Podcast

[In this show](https://www.youtube.com/@TheInsideView), Michaël Trazzi discuss why experts believe what they believe about the future of AI progress. You can listen to the audio recordings and read the transcripts at [theinsideview.ai](https://theinsideview.ai).

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




The Inside View

The [AI Safety Ideas platform](https://aisafetyideas.com/) is a public Apart tool used for collecting research in AI safety, collaborate on projects, and facilitate mentorship.

## Team

The [Apart Research](https://apartresearch.com/ "‌") team, we have multiple volunteers and collaborators that have joined on our [Discord](https://apartresearch.com/join "‌") and in our other projects.

## Open impact metrics of AI Safety Ideas

The AI safety research ideas platform commits to transparency and sharing our metrics. We hope this enables easier evaluation of the platform's impact. Read the [forum post](http://forum.effectivealtruism.org/posts/nxeL7XFvtQCx7hpxo/everyone-show-us-your-numbers "‌") for more details.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




AI Safety Ideas

A weekend of intense, fun, and collaborative research on the most interesting questions of our day from machine learning safety. With speakers from top institutions and companies in artificial intelligence development, you will get a chance to work on impactful research.

Alignment Jams happen across the world in:

- USA
- EU
- UK
- Mexico
- Brasil
- India
- Vietnam
- Israel
- Australia and
- Canada

[Website](https://alignmentjam.com/ "")

[Next hackathon](https://alignmentjam.com/jam/benchmarks "‌")

[Discord community](https://discord.gg/dYUWDm7Ben "‌")

[Add to your calendar](https://calendar.google.com/calendar/u/0?cid=ZjViYmMzNjlhNDFmZjg5MmY5ZTkxOWJjMGVkMmFlNjRmOTBjMmVjNTMzZDVhMTZhMWNkMjY4ZjU1M2JhMTBlY0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t "‌")

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]

Alignment Jam

[Apart Lab](https://apartresearch.com/#Lab) collaborates with scholars in AI safety to publish technical research into safer and more aligned machine learning systems.

## Recent accepted papers from the Apart Lab

### Neuron to Graph: Interpreting Language Model Neurons at Scale

Alex Foote, Neel Nanda, Esben Kran, Ionnis Konstas, Shay Cohen, Fazl Barez

[Read paper](https://arxiv.org/abs/2305.19911 "‌") | [Visit project site](https://n2g.apartresearch.com/ "‌")

May 5, 2023  |  RTML workshop at ICLR 2023

### Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark

Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ionnis Konstas, Fazl Barez

[Visit project site](https://specificityplus.apartresearch.com/ "‌")

July 10, 2023  |  ACL 2023

Apart Lab

Videos about Artificial Intelligence Safety Research, for everyone.

AI is leaping forward right now, it's only a matter of time before we develop true Artificial General Intelligence, and there are a lot of different ways that this could go badly wrong for us. Putting aside the science fiction, this channel is about AI Safety research - humanity's best attempt to foresee the problems AI might pose and work out ways to ensure that our AI developments are safe and beneficial.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]

Robert Miles AI Safety

The Machine Intelligence Research Institute is a research nonprofit studying the mathematical underpinnings of intelligent behavior. Our mission is to develop formal tools for the clean design and analysis of general-purpose AI systems, with the intent of making such systems safer and more reliable when they are developed.

The field of AI has a reputation for overselling its progress. In the “AI winters” of the late 1970s and 1980s, researchers’ failures to make good on ambitious promises led to a collapse of funding and interest in AI. Although the field is now undergoing a renaissance, overconfidence is still a major fear; discussion of the possibility of human-equivalent general intelligence is still largely relegated to the science fiction shelf.

At the same time, researchers largely agree that AI is likely to begin outperforming humans on most cognitive tasks [in this century](https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ "‌"). Given how disruptive domain-general AI could be, we think it is prudent to begin a conversation about this now, and to investigate whether there are limited areas in which we can predict and shape this technology’s societal impact.

Researchers at MIRI tend to be relatively [agnostic](https://intelligence.org/faq#imminent "‌") about how the state of the art in AI will change over the coming decades, and how many years off smarter-than human AI systems are. However, we think some qualitative predictions are possible:

- As perception, inference, and planning algorithms improve, AI systems will be trusted with increasingly complex and long-term decision-making. Small errors will then have larger consequences.
- Realistic goals and environments for general reasoning systems will be too complex for programmers to directly specify. AI systems will instead need to inductively learn correct goals and environmental models.
- Systems that end up with poor models of their environment can do significant harm. However, poor models limit how well a planning system can control its environment, which limits the expected harm.
- There are fewer obvious constraints on the harm a system with poorly specified goals might do. In particular, an autonomous system that learns _about_ human goals, but is not correctly designed to align its own goals to its best model of human goals, could cause catastrophic harm in the absence of adequate checks.
- AI systems’ goals or world-models may be brittle, exhibiting exceptionally good behavior until some seemingly irrelevant environmental variable changes. This is again a larger concern for incorrect goals than for incorrect belief and inference, because incorrect goals don’t limit the capability of an otherwise high-intelligence system.

Stuart Russell, a MIRI research advisor and co-author of the [leading textbook on artificial intelligence](http://aima.cs.berkeley.edu/ "‌"), argues in “[**The Long-Term Future of Artificial Intelligence**](https://www.youtube.com/watch?v=GYQrNfSmQ0M "‌")” that we should integrate questions of robustness and safety into mainstream capabilities research:

> Our goal as a field is to make better decision-making systems. And that is the problem. […If] you’re going to build a superintelligent machine, you have to give it something that you want it to do. The danger is that you give it something that isn’t actually what you really want — because you’re not very good at expressing what you really want, or even knowing what you really want — until it’s too late and you see that you don’t like it.
>
> If you think about it just in terms of an optimization problem: The machine is solving an optimization problem for you, and you leave out some of the variables that you actually care about. Well, it’s in the nature of optimization problems that if the system gets to manipulate some variables that don’t form part of the objective function — so it’s free to play with those as much as it wants — often, in order to optimize the ones that it is supposed to optimize, it will set the other ones to extreme values.
>
> My proposal is that we should stop doing AI in its simple definition of just improving the decision-making capabilities of systems. […] With civil engineering, we don’t call it “building bridges that don’t fall down” — we just call it “building bridges.” Of course we don’t want them to fall down. And we should think the same way about AI: of course AI systems should be designed so that their actions are well-aligned with what human beings want. But it’s a difficult unsolved problem that hasn’t been part of the research agenda up to now.
>
> We want to change the field so that it feels like civil engineering or like nuclear fusion. [… We] created a hydrogen bomb explosion — unlimited amounts of energy, more than we could possibly use. But it wasn’t in a socially beneficial form. And now it’s just what fusion researchers do — containment is what fusion research _is_. That’s the problem that they work on.

In line with Russell’s talk, MIRI’s work is aimed at helping jump-start a paradigm of AI research that is conscious of the field’s long-term impact. Our [methodology](https://intelligence.org/2015/07/27/miris-approach/ "‌") is to break down the alignment problem into simpler and more precisely stated subproblems, develop basic mathematical theory for understanding these problems, and then make use of our newfound understanding in [engineering](https://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/ "‌") applications.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Machine Intelligence Research Institute

Our vision is a positive and secure future. Pour Demain is a non-profit think tank that develops policy proposals on neglected issues positively impacting Switzerland and beyond. We are committed to fact-based policies, developing scientifically sound recommendations and facilitating the exchange between science, politics and civil society.

## What does Pour Demain do exactly?

In dialogue with science, politics, and civil society, Pour Demain develops recommendations relating to neglected topics such as pandemic prevention or artificial intelligence (AI).

Examples:

- [Pandemic early warning system](https://en.pourdemain.ch/post/early-warning-system-against-pandemics-will-save-up-to-chf-30-billion "‌")
- [Monitoring the opportunities and risks of AI](https://en.pourdemain.ch/post/ai-monitoring-op-ed-in-the-neue-z%C3%BCrcher-zeitung "‌")

To this end, the think tank maintains a lively exchange with collaborators from science, civil society, politics, and administration. We also organise workshops with experts and public events.

## How does Pour Demain prioritise recommendations?

Pour Demain evaluates the recommendations in a structured process. The focus is on assessments by experts, from professional articles to interviews to written expert surveys, which Pour Demain conducts.

The Pour Demain team applies the following criteria in the assessment of new recommendations (simplified):

- "Do no harm" - exclusion criterion: Recommendations with potential harm are excluded.
- Impact: How positive is the impact of the recommendation on Switzerland and beyond?
- Implementability: How likely is it that the recommendation will be implemented? How long is it likely to take for the implementation of the recommendation to have an impact?
- Neglected: How neglected is the recommendation? What contribution can Pour Demain make compared to other actors?

The think tank follows an iterative approach. Recommendations are not static, but are continuously adapted to new circumstances. We regularly re-evaluate the recommendations with experts and citizens.

## What is the role of the Pour Demain team?

We mediate between science, politics and civil society. In this role, we have a certain expertise in all areas, but we do not see ourselves as experts in all topics.

We draw on the expertise of our network of over 200 professionals and the approximately 25-member advisory board. The expertise is always contextualized with regard for the concerns of the Swiss population and civil society.

As a team we act independently or self-organised.

## Is Pour Demain independent? Who funds the think tank?

Pour Demain is a non-profit, tax-exempt, independent association. It is financed through charitable donations, e.g. from private individuals or the Hasler Foundation.

The donations are independent of corporate interests or party political preferences.

The definition and implementation of our activities is independent of donors.

Pour Demain refrains from paid or unpaid mandates.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]


Pour Demain

Steering transformative technology towards benefiting life and away from extreme large-scale risks.

We believe that the way powerful technology is developed and used will be the most important factor in determining the prospects for the future of life. This is why we have made it our mission to ensure that technology continues to improve those prospects.

![conference1501041-2-scaled.jpg](https://futureoflife.org/wp-content/uploads/2015/10/conference1501041-2-scaled.jpg)

Since its earliest days, human civilisation - and through it increasingly all life on Earth - has been shaped by technology. In the current decades, however, something unprecedented is happening. Humanity now holds the power to affect, and even destroy, life on Earth as a whole. And recently, through the continued development of biotechnology and machine intelligence, we have entered an era in which life and intelligence will be engineered by life and intelligence, rather than by evolution. The rapidly increasing power of these technologies implies that the changes of the next several decades, and their effect on the long-term future of life, are likely to be profound - and perilous.

**The mission of the Future of Life Institute, established in 2015, is to steer transformative technology towards benefitting life and away from extreme large-scale risks.**

In pursuit of this mission, we support the development of institutions and visions necessary to manage world-driving technologies and enable a positive future. And we seek to reduce large-scale harm, catastrophe, and existential risk resulting from accidental or intentional misuse of transformative technologies.

Towards these aims, FLI engages in a number of initiatives and strategies, including:

- **Policy development and advocacy** to bridge the gap between the experts who understand transformative technologies and the public institutions with the legitimacy and means to govern them.
- **Outreach and education** to help policymakers, technologists and the general public understand the challenges and opportunities we face. This includes envisioning positive futures as an antidote to prevailing defeatism, and formulating the interventions we might need now if we're to steer toward such futures.
- **Research and research support** because managing transformative technologies will require research on problems and in fields that existing sources have not sufficiently supported through grantmaking or otherwise.
- **Grantmaking** to support research on problems and in fields stemming from transformative technologies that are otherwise insufficiently resourced.
- **Institution-building** to design, launch, and support new organizations and agreements to improve the governance of transformative technologies.
- **Convening and coordinating** events and activities, because these issues are global and addressing them will require large-scale coordinated action, even amongst rivals.

FLI presently focuses its energy on issues of advanced artificial intelligence, militarized AI, nuclear war, and new pro-social platforms. We also pursue topics in other fields including bio-risk and the preservation of biodiversity. However, we consider our purview to be as wide as necessary to pursue our core mission.

In carrying out its activities, FLI is committed to a number of core principles:

- **Impact-driven:** We take on projects because in our assessment we anticipate they will have large positive impact, both now and in the longer term.
- **Cognizant of urgency:** Analysis paralysis should not indefinitely forestall action which is based on the best available information. Key events in human history are transpiring, and people and the Earth are being harmed right now.
- **Forward-thinking and anticipatory:** While recognizing that some aspects of the future are difficult or impossible to forecast, many problems and opportunities can be anticipated and must be planned for before they are fully manifest.
- **Driven by science and reason:** FLI aims to act on humanity's hard-won understanding of both how the world functions, and how we can come to better understand how it functions.
- **Inclusive:** In our work we seek to honour the experiences, respected rights, and autonomy of the world's people and other sentient beings. FLI believes that technology, social structures and governments are instrumental: they are meant to serve us and not to be served. In contrast, people are not instrumental, which means that positive impact on the world should be achieved while maintaining integrity, kindness, and respect for others.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]

Future of Life Institute

CHAI is a multi-institution research group based at UC Berkeley, with academic affiliates at a variety of other universities.

### Mission

CHAI’s goal is to develop the conceptual and technical wherewithal to reorient the general thrust of AI research towards provably beneficial systems.

Artificial intelligence research is concerned with the design of machines capable of intelligent behavior, i.e., behavior likely to be successful in achieving objectives. The long-term outcome of AI research seems likely to include machines that are more capable than humans across a wide range of objectives and environments. This raises a problem of control: given that the solutions developed by such systems are intrinsically unpredictable by humans, it may occur that some such solutions result in negative and perhaps irreversible outcomes for humans. CHAI’s goal is to ensure that this eventuality cannot arise, by refocusing AI away from the capability to achieve arbitrary objectives and towards the ability to generate provably beneficial behavior. Because the meaning of beneficial depends on properties of humans, this task inevitably includes elements from the social sciences in addition to AI.

Find out more about the people who work with and at CHAI [here](https://humancompatible.ai/people "‌").

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]

Center for Human-Compatible AI

We are an interdisciplinary research centre within the University of Cambridge dedicated to the study and mitigation of existential risks

## Existential risks in the 21st Century

Now is an important time for efforts to reduce existential risk. There are new and largely unstudied risks associated with powerful emerging technologies and the impacts of human activity, which in the worst case might pose existential risks. We want to reap the enormous benefits of technological progress while safely navigating these catastrophic pitfalls. These threaten everyone – we can only tackle them together.

[GET IN TOUCH](https://www.cser.ac.uk/contact/ "‌")

![54d317dbff5c3647b426dc13b39e6cef.jpg](https://www.cser.ac.uk/media/cache/54/d3/54d317dbff5c3647b426dc13b39e6cef.jpg)

## Our primary aims

### Reduce risk

We aim to reduce the risk of human extinction or civilizational collapse.

### Study risks

We work to understand extreme risks associated with emerging technologies and human activity; and to develop a methodological toolkit to aid us in identifying and evaluating future extreme risks.

### Collaboration

We develop prevention and mitigation strategies in collaboration with academics, industry and policymakers, so we can gain the benefits and avoid the risks of technologic progress.

### Community

We foster a reflective, interdisciplinary, global community of academics, technologists and policymakers examining individual aspects of existential risk and coming together to integrate their insights.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]




Centre for the Study of Existential Risk

This project aims to improve our understanding of the likely impacts of human-level artificial intelligence.

The intended audience includes researchers doing work related to artificial intelligence, philanthropists involved in funding research related to artificial intelligence, and policy-makers whose decisions may be influenced by their expectations about artificial intelligence.

**The focus is particularly on the long-term impacts of sophisticated artificial intelligence.** Although human-level AI may be far in the future, there are a number of important questions which we can try to address today and may have implications for contemporary decisions. For example:

- What should we believe about timelines for AI development?
- How rapid is the development of AI likely to be near human-level? How much advance notice should we expect to have of disruptive change?
- What are the likely economic impacts of human-level AI?
- Which paths to AI should be considered plausible or likely?
- Will human-level AI tend to pursue particular goals, and if so what kinds of goals?
- Can we say anything meaningful about the impact of contemporary choices on long-term outcomes?

Today, public discussion on these issues appears to be highly fragmented and of limited credibility. More credible and clearly communicated views on these issues might help improve estimates of the social returns to AI investment, identify neglected research areas, improve policy, or productively channel public interest in AI.

#### The goal of the project is to clearly present and organize the considerations which inform contemporary views on these and related issues, to identify and explore disagreements, and to assemble whatever empirical evidence is relevant.

The project is provisionally organized as a collection of posts concerning particular issues or bodies of evidence, describing what is known and attempting to synthesize a reasonable view in light of available evidence. These posts are intended to be continuously revised in light of outstanding disagreements and to make explicit reference to those disagreements.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]

AI Impacts

_GCRI’s mission is to develop the best ways to confront humanity’s gravest threats._

The Global Catastrophic Risk Institute (GCRI) is a nonprofit, nonpartisan think tank. GCRI works on the risk of events that could significantly harm or even destroy human civilization at the global scale. As a think tank, GCRI bridges the world of scholarship and the world of professional practice in government, private industry, and other sectors. We aim to develop highly effective solutions for reducing the risk by leveraging both the best available scholarship and the demands of real-world decision-making.

GCRI was founded in 2011 by [Seth Baum](http://sethbaum.com/ "‌") and [Tony Barrett](http://tony-barrett.com/ "‌"). The two met at a conference of the Society for Risk Analysis (SRA), the leading academic and professional society for all aspects of risk. GCRI continues to be involved in SRA, and we also play a central role in an international community of catastrophic risk scholars and practitioners.

**Annual Reports**

At the end of each calendar year, GCRI creates an annual report dedicated to recent accomplishments and future plans. Below are GCRI’s most recent annual reports:

- [2022 Annual Report](https://gcrinstitute.org/2022-annual-report/ "‌")
- [2021 Annual Report](https://gcrinstitute.org/2021-annual-report/ "‌")
- [2020 Annual Report](https://gcrinstitute.org/2020-annual-report/ "‌")
- [2019 Annual Report](https://gcrinstitute.org/2019-annual-report/ "‌")
- [2018 Annual Report](https://gcrinstitute.org/2018-annual-report/ "‌")

**Financial Information**

GCRI is a project of the fiscal sponsorship organization [Social and Environmental Entrepreneurs (SEE)](http://www.saveourplanet.org/ "‌"). SEE is a nonprofit public charity exempt from federal income tax under Section 501(c)(3) of the Internal Revenue Code. [Your donation](https://gcrinstitute.org/donate "‌") is fully tax deductible.

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]

Global Catastrophic Risk Institute

Giving to the right charities means you can save hundreds of lives. We help you find them. Why do we do it? Because we strive for a world where the worst problems are solved as soon as possible.

## Effective Giving Guide

We’ve gathered a lot of information on what it means to give effectively and how to have the biggest impact through your philanthropy. Learn about our approach in our three-part guide on effective giving - from picking a worthy cause to finding the right charity.

[Read the guide](https://reg-charity.org/effective-giving-guide "‌")

## Donation Advice

Picking the right charity is a difficult task for anyone working independently. This is where we come in. We examine the growing body of research around cost-effectiveness and distill this information into donation advice for individuals and organizations.

[Receive more information](https://reg-charity.org/donation-advice "‌")

## Charity Recommendations

We provide a public list of outstanding charities for donors to support. We strongly believe that any contribution to these organizations will make a significant positive difference in the world. The list is updated regularly to reflect new developments and evidence.

[Look at the charities](https://reg-charity.org/recommended-charities "‌")

[This project was created by the Impact Markets team. Please visit the website of the organization for more information on their work.]
