update March 2021: as conversations around this evolved the focus has narrowed to building novel forms of marketplaces that maximize the funding they generate for their complementary public goods. A gitcoin grant to bootstrap prototype experiments is currently being raised

– (original post)

How to set up good incentives for information that guides choosing the best action is a crucial challenge for collective cognition. It has received mostly scholarly attention on the literature around Decision Markets. Decision markets seek to both predict and decide the future. They differ from more traditional markets such as betting (“prediction markets”) in that they seek to influence the object of their attention instead of merely being influenced by it.

Robin Hanson in Futarchy proposes to vote on values, bet on beliefs. While superficially eloquent this works better as alliteration than as an incentive alignment. If one tries to use betting mechanisms to guide decisions things do not work out. Betting on the value of the reward conditional on each potential action and voiding bets for unchosen actions is not, as Othman and Sandholm pointed out, incentive compatible when the decision is made by selecting the action with the highest expected reward as determined by the market. This family of mechanisms require that strictly dominated actions be taken with substantial positive probability (Chen et al). These theoretical difficulties are matched by a lack of practical implementations.

Advice Auctions may be the mechanism that enables market driven decision making. It is based on an old and simple idea, giving your advisors a share of the reward, with an algorithmic twist of facilitating the advisors themselves seeking advice in the same way recursively. The space these markets work over is explcitly not expectations over actions but instead the selection of a good policy for making them. This makes auditing the advice posible, as well as evaluating it’s counter-factual performance.

Recently Osterheld and Conitzer show a fundamental limit to elicitation for decision making, only the best action and it’s expected reward are elicitable. Advice auctions match this precisely, further they are able to deal with complementary information distributed between different experts, something that betting mechanisms used prediction markets fail at.

In the spirit of Constructive Economics the focus is on making it real. Smart contracts running on proof-of-stake blockchains provide a playground for experimenting with novel economic structures. A self-referential experiment in computational social science: a DAO run by, and dedicated to the development of, practical designs for incentivizing advice that leads to better choices. The DAO itself making decisions using the proposed mechanisms.

The goal is to experiment with what applications are well suited to these mechanisms, how the mechanisms need to be refined, and what is needed to make this happen. Potential applications:

• choosing if a seed stage fundaraising team/project should be funded
• choosing profit maximizing bid-ask spreads for automated market makers, more broadly choosing parameters of control systems in governance minimized systems.
• choosing suppliers when quality is hard to observe ex-ante but affects an ex-post observable reward
• choosing what to learn to maximize market income in some future period

After some conversations about advice auctions and related work on decision elicitation, at the DIMACS Workshop on Forecasting: From Forecasts to Decisions and conversations witht he open research collectives aroun the Ethereum ecosystem, there seems to be enough momentum to have a open research collective for research and design on advice and policy elicitation.

Some of the initial topics of the research are around how to leverage the different parts of the literature on elicitation:

• Decision markets are to prediction markets, as bandit problems are to fully supervised ones. Data from related supervised tasks can be valuable in training models that are fine tuned on a bandit or reinforcement learning enviorment. How can prediction markets for variables related to those of the choice but that do not directly stem from it be used to improve the elicitation of the choice.

• How can the share of the reward that is auctioned off be set? The basic tradeoff is that making the share close to 1 reduces the downside risk at the cost of also reducing the upside for the decision maker. Higher values also make the capital requirements for participation, effectively limiting competition between advice providers.

• What is the role of meta-questions as in bayesian truth serum and related mechanisms?

• Would giving away a token to particularly skilled groups of potential advisors (say for example high scoring users in forecast competition sites)? Should it be possible to cash them out after some amount of trading only (so they are not fungible until they have acumulated some history and then they could be transformed into the standard fungible version).

If youd like to get involved, reach out.