Toward Correctly Trusting AI
Not that I think we can entirely...
I wrote this a month or so ago with for the EA Automating Wisdom Contest as One of Two Essays Towards Correctly Trusting AI. Turns out we both submitted copies with all of our ridiculous notes… very demure, very don’t-give-a-fuck of us. I’m posting part of this essay here (thanks for the email limits Substack), the rest you can read here, but I’m doing this because I’m submitting another essay to a different AI essay contest, tomorrow and well, posterity.
Enjoy.
What can be pursued from a CEO’s desk? The notion that the profit motive is the sole function of today’s corporate governors is often implied, but I suggest, like Timothy Williamson’s, opening line of Philosophy of Philosophy (“What can be pursued in an armchair?”) that because optimal business functions can be done from the CEO’s desk, doesn’t mean that they must, nor does it mean that the CEO can’t also pursue optimal outcomes beyond the scope of the desk. Machine learning advances are not solely driven by corporate utility, nor is our understanding of the world purely from academic science and philosophy. Combined efforts reveal mounting catastrophic risks that need further integration to mitigate. In academic labs and tech R&D, scientists think like CEOs and developers like philosophers, but few actual philosophers are involved in this industry. Philosophy needn't be confined to armchairs, nor the CEO to her desk. Creating wise AI and systems requires the collaboration of CEOs, philosophers, and everyone in between.
Wise AI must be made by wise humans through wise systems
The current state of corporate governance may be a significant factor in what some call the metacrisis and the underpinnings of many existential risks not only in the development of AI, but also in the potential for subjugation through perverse economic incentives in campaign finance and legislative practices, climate-related catastrophes political perpetuating and exacerbating extreme socioeconomic inequalities, and others. Producing “wise AI” may mitigate all of these risks, but this would also mean we’ve at least started planting seeds to reduce them ourselves, and this will likely require improving the corporate alignment problems. Here I make specific prescriptions for creating an ecosystem that bridges epistemological and philosophical gaps in the development of wise human/AI systems. Human and machine deliberation must be done in parallel, this is my recommendation for how to “correctly” trust AI.
The argument for my argument is made in detail in Anton Greenaway’s submission and the prescriptions that follow serve as potential implementations of what Owen Cotton-Barratt’s reflective governance. We take this a step further to say reflexive governance might be more suited; the distinction is that reflection may help stakeholders learn and change, but also may unveil a paradox - that basic reflection may be self-defeating in an organization like a corporation (more in Prescription 4) For example, a reflection committee might find that it can’t immediately justify the resources allocated in the time spent, and dissolve itself accidentally to serve a singular goal of the organization (e.g., profit), whereas a reflexive committee would only do this with intent and sound justification from a variety of stakeholders. AI development needs deeper and more adaptable reflexion to question assumptions that may be taken for granted.
This is possible under the corporate dictum, which contrary to popular belief is no longer synonymous with the Friedman shareholder doctrine (the sole responsibility of a corporation is to its shareholders), however stock prices don’t necessarily reflect changes in hearts and minds of corporate strategists. One way to move the market towards creating wise AI is to take a page out of Henry Ford’s book. Pre-empting FDR’s Fair Labor Standards Act, Ford drastically improved labor standards while simultaneously creating a market for the company’s products. Nearly a century later, corporate leaders in AI have a similar opportunity to improve existential risks and create a sound environment for ethical and human-centered AI development. Some leaders (69%) who have yet to understand the value of responsible AI policy may find these actions counterintuitive, but whether you look at them as beneficial for shareholders or all stakeholders, incorporating philosophical discussion and inquiry into AI development at the corporate level can not only provide an overall societal benefit and
The market's attempts at early self-governance for responsible philosophical governance of AI can be likened to Ford's introduction of the 40-hour workweek, representing a significant shift towards balancing innovation with ethical and practical considerations. Ford recognized that excessive working hours were detrimental to both productivity and employee well-being, thereby establishing a sustainable model for industrial labor. Similarly, today's AI landscape faces the imperative to incorporate rigorous philosophical frameworks to address ethical dilemmas, biases, and societal impacts inherent in advanced AI systems. Just as Ford's policy set a new standard that enhanced productivity and worker satisfaction, the integration of comprehensive ontological and ethical principles in AI development aims to ensure that technological advancements are aligned with human values and social good, fostering trust and long-term viability in the AI market. This analogy underscores the necessity of a structured, principled approach to managing transformative innovations in both historical and contemporary contexts.
To create an ecosystem that takes wise human/AI seriously, we might consider modeling some of the practices that have led to significant advances in medicine. Anthropic’s AI Safety Levels (ASL) already nod to biomedical advances, the result of integrating academic and corporate goals for discovery and profit. However, to date, AI companies have not faced the significant regulatory intervention of the biomedical industry, which has added significant cost, time, and (sometimes) unnecessary bureaucracy to the development of medicines. Professor Daniel Carpenter makes the argument that the Food and Drug Administration has become the most powerful regulatory agency in the world through a succession of legislative acts and public health crises (e.g., thalidomide) that allowed for the expansion of biotech and is currently developing the pitfalls and potential of applying an approval-regulation model to AI products. However, there are several assumptions in developing medicinal products that are not baked into developing tech products, one is that the observational effects of molecules can be measured easier than multifunctional digital tools and products. How does one study the effects of mixing saltpeter, sulfur, and charcoal (gunpowder) on society? We can speculate as to how a Chinese Alchemical Regulatory Agency might have functioned at the time, but only by an integration of social science, metaphilosophy, and historical analysis can we reach any decent prescriptions. Tech may not warrant the same regulation as biotech, but there may be some shared principles.
To address power imbalances in the American political system, we should create an ecosystem that considers all stakeholders not just those with the economic influence. Average consumers lack recourse to the detrimental effects of tech, and there is currently no incentive for companies developing these products to investigate downstream harms. Unlike biomedical advances, the potential harms of tech development are less visible, leading to lagging regulation. This presents a unique opportunity to demonstrate the benefits of industrial self-governance before traditional regulatory agencies impose mandates.
Philosophy of medicine, rooted in Jeremy Bentham’s utilitarianism, has focused on developing sound science at the expense of phenomenology, ontology, other epistemologies, and perhaps even the philosophy of science itself. Persistent questions in medicine include the nature of mental disorders how to balance subjective patient experience with objective data collection, managing conflicting data defining sickness or even what it is to be alive, and the social impacts of cosmetic or elective procedures. With GATTACA-level gene editing technology around the corner, one would hope we'd have thought about these things by now. In tech, the existence of this essay competition lends some hope for better approaches with AI.
Insofar as there is a philosophically “good” way to go about developing AI, it is likely to require more the development of wise or “good” AI principles (i.e., Mary Migdley’s philosophical plumbing). For more, see Anton Greenaway’s submission. The prescriptions herein are predicated on the idea that establishing good conceptual hygiene is essential for this endeavor:
Develop and fund a Wise AI Institute
Create a philosophical/technical job pipeline
Operate based on a set of Wise AI principles
Iterate governance and principles
Prescription 1: Developing Wise AI Institute
What’s considered wisdom on cybernetic activities is currently discussed between academics and what I call ‘neoacademics’. Neoacademics are often auto-didactic creatives, artists, writers, thinkers, intellectuals, start-up founders, developers, policy drafters, and independent researchers - not necessarily professional academics. They have a deep interest in knowledge itself and seek opportunities to increase their own, and that of others. Traditional academic careers don’t allow neoacademics the breadth and/or relative depth that they long for in their independent studies and pursuits. The career path is incredibly (and ironically)narrow, and in some cases, careers in academia pay below the poverty line. So, one shouldn't expect the most integrative thinkers to be there. The democratization of knowledge and access to scholars and self-publishing channels have allowed neoacademics to contribute significantly to scholarly conversation and research without formal training or an academic pedigree, the founder of Less Wrong, being a prime example.
In the short-term, a society that values innovation, wisdom, and collective intelligence would seek to harness the power of these kinds of thinkers and integrate them with more traditional thought leaders in academia, Non-Governmental Organizations, government, and industry. Emphasizing diverse career origins and paths would foster significant diversity of thought and highly-capable team formation around important societal problems. A broad knowledge base and wide variety of expertise will be important as the gap between current machine learning and something like Artificial General Intelligence closes. Creating such an institute could hold long-term potential for developing new epistemological and educational classes of organizations that value complexity.
The initial function of such an institute would be to implement the following prescriptions, and given the 4th prescription’s advocacy of reflexive iteration, the institute’s functions may shift as needed.
The structure and governance of a Wise AI Institute would be important, but there are various problems with all governance models. It could grow out of a for-profit initiative and serve as a non-profit arm, but an accountable board and executive director are paramount. To maximize utility across academia, government, and industry, the institute should offer grants, research support, regulatory and policy guidance, and commercial support mechanisms related to society, cybernetics, and philosophy.
Cybernetics, despite its intimidating connotations, saturates our lives with feedback loops, intentional or not. We are also at an impasse in many cultural realms where philosophies and epistemologies seem incommensurable, so such an institute would need to take a careful look at the utility and verisimilitude of pluralism. Despite decades of mounting support for scientific pluralism since Paul Feyerabend, many rationalists find this kind of discussion unappealing, or even diseased, so particular care will need to be taken with terminology. Using the word ‘cybernetic’ offers a bit of a shock value here, and serves to demonstrate the value and necessity of messaging. One of my central critiques of EA is the straw-Vulcan aesthetic not to mention the general cluelessness we all face. It’s a marketing critique. I understand lukeprog’s desire for “clear, rigorous, and scientifically respectable” philosophy, but esoterica, marketing, and aesthetics also have a place - especially concerning information hazards. Machiavelli got this centuries ago, it’s time rationalists see that straight and narrow isn’t the best or only way forward.
This is why we need to invite a broader cross-section of philosophers and house them where they don’t feel subordinate to rationalists. Forming an institute that prioritizes philosophy, would help create a conducive environment where thinkers of all types could focus on relevant problems.
Prescription 2: Create a philosophical/technical job pipeline
A society that values the development of wise AI would develop a respective industry if that society values industry as we currently do. Such an industry would learn from mistakes in other industries in an attempt to model best practices that have developed.
Since the 1880s, pharmaceutical companies have employed scientists, manufacturers, and salespeople. In 1962, the Kafauever-Harris Amendment to the Federal Food, Drug and Cosmetic Act required drug manufacturers to prove the efficacy and safety of new drugs before marketing. To navigate this, in 1967, Upjohn Pharmaceuticals created a technical/commercial hybrid role called Medical Science Liaisons (MSL). MSLs became a technically adept field force that could communicate with medical-technical expert customers (i.e., physicians) in the field and navigate a complex regulatory environment. This role helped disseminate medical information and increase the overall working knowledge of the industry. Ultimately, a boom of jobs for clinical trialists, regulatory experts, and bureaucrats developed after market regulations, but early anticipation of such an explosion by the industry itself could have saved lives, and resources and accelerated societal integration of medical advances.
The cat-and-mouse game of market vs regulation creates red tape, ultimately increasing the cost of drugs through pharma’s expenditure in R & D, which has been long-suspected of covert marketing allocations through functions like the MSL - which only leads to further scrutiny and regulation. The MSL role was largely a commercial endeavor, and without a code of ethics, can fall victim to unethical marketing practices through “off-label” promotion, where a product is approved for a specific use, but intentionally marketed for another. For example, when I started as an MSL at Genzyme in 2014, there was enhanced scrutiny around my team because the company was under a Corporate Integrity Agreement after a bioresorbable adhesion barrier product had been promoted without regulatory approval and was being dissolved into a saline “slurry” and painted on skin around a surgical site. Lore has it that sales reps and maybe even some MSLs were hosting workshops on how to do this before someone blew the whistle. Genzyme agreed to pay $32.5M in fines, and individual employees faced criminal charges. By incentivizing “good” behavior in sales and MSL functions, Genzyme could have saved itself a lot of trouble
The tech industry is largely unregulated from an efficacy & safety standpoint, but this is unlikely to continue. Accountability and transparency of algorithmic and autonomous systems, cybersecurity, data privacy, ethical applications of human-AI interactions are all possible avenues for regulatory intervention. Current “AI” companies fall into two categories: 1) Organizations trying to improve on existing machine learning (ML) technology and 2) organizations (claiming) to use existing ML technology to improve processes outside the field. Even if the aforementioned regulatory avenues are not pursued, the Federal Trade Commission (FTC) is already enforcing false claims of AI-driven technology in the latter. There is a clear need for wisdom in tech, especially as it pertains to marketing, commercialization, and educating consumers.
Regulatory action is looming, and may or may not help the development of wise AI, as regulation can have variable scope, compliance, and enforcement, as well as significant effects on innovation, public awareness, and consumer legislation. Building the early groundwork for teams that will be able to navigate complex regulatory environments will be essential to shaping and managing potential audits, and product submissions for regulatory approval.
I recommend a Technical Theoretical Liaison (TTL) role for the dual purpose of education and philosophical discussions. Like the role of the MSL, the TTL would be largely an educational role, but as much as pharma leadership will deny this, the MSL serves an inherent commercial function just by being customer-facing. The Seprafilm incident at Genzyme may have been mitigated by a shared governance of this function, and increased interface with “wise” practices within the company. Currently, the role of the MSL is enigmatic for many companies. They’re a non-revenue generating field force that facilitates regional and national educational programs, interfaces with thought leaders, and submits and evaluates those thought leaders’ grant proposals at the company. To incentivize pushing forward on “related but less important questions” as per the announcement request for this contest, I can see this playing out two ways with a hybrid possibility:
Wise AI-Funded TTLs
Company-Funded TTLs
Hybrid
In Scenario 1, Technical Liaisons would be employed by the third-party Wise Institute. They would not have access to proprietary information but would serve as community liaisons with industry-wide educational goals. The primary benefit of this scenario would be that TTL’s would be independent actors, with a broad knowledge base of many products, technologies, and philosophical frameworks, with the singular priority of developing industry-wide wise practices, however, the lack of insight into company practices would diminish the capacity for industry-wide self-governance and potential whistle-blowing. Scenario 2 would be more akin to the traditional MSL role in pharma. The benefit: TTL’s have near total information about a singular product; The detriment: TTL’s expertise is “bought” by a single company and industry-wide wise practices become a lower-ranking priority behind company objectives. Scenario 3 is what I would prefer to see, but it may be the hardest to implement. In this instance, companies and a Wise AI Institute would both pay a portion of the TTL’s salary and contribute to a joint educational fund to facilitate some industry-wide educational objectives. They would also be able to fund their own activities and grants. TTL’s would be incentivized to maintain a more balanced perspective on industry behaviors, but I foresee some difficulty managing proprietary knowledge.
It’s been estimated that were are only around 400 AI safety researchers in 2022, and this number is soon to double, but as of yet, there is no field-based educational role in the tech industry. It would be a high-level position that requires a technical understanding at a Ph.D or master's level with research in philosophy or ML owing to the depth of the literature. MSL teams are multidisciplinary with shared expertise from Ph.Ds, pharmacists, nurses, and physicians. It also requires interpersonal skills for relationship building. Each company hires and employs its own field force to execute strategy by sending liaisons to meet with industry leaders to discuss relevant company data and field-wide studies. This works well for companies, and has created high-paying jobs, but the role is not well-looked upon by academics. If I were to go back 60 years, I’d suggest that each company pay into an industry-wide pool that ensures good practices for all liaisons and incentivizes “wise” actions. This might elevate the role and allow MSLs to have a more independent meta-research function within their domain. TTLs could do this easily given the capacity for self-publishing.
Small to mid-size research grants like the current submission and more technical programs are essential for developing interest, but without significant longer-term infrastructure, wise AI will fall to the wayside. Said infrastructure must support product and technology development, conveying benefit to all stakeholders and ensuring enthusiasm about the project.
Ideally, functionalities of a wise AI pipeline would be dynamic and agreed upon by stakeholders in the industry rather than imposed upon them, but ultimately, we are likely to enact and adopt regulations. Just as we are likely to co-evolve our own programming alongside that of the machines we create, we are likely to evolve alongside a regulatory framework. Physicist Ursula Franklin notes that “Today's real world of technology is characterized by the dominance of prescriptive technologies… prescriptive technologies … have raised living standards and increased well-being. At the same time they have created a culture of compliance. The acculturation to compliance and conformity has, in turn, accelerated the use of prescriptive technologies in administration, government, and social services. The same development has diminished resistance to the programming of people.” By creating prescriptions (even in this document) we diminish our resistance to programming, which is why it’s important to start this pipeline that will develop people to:
Prescription 3: Operate based on a set of Wise AI principles
A Wise AI Institute might concern itself with two audiences: an intra-industry audience and a public audience. I’d initially place more emphasis on the intra-industry goals because there are other philosophical origanizations concerned with public education, and even some in the industry. Effective Altruism offers entry-level educational programs, but these programs place (some might say) inordinate emphasis on utilitarian ethics. Tactics like a public awareness campaign might be useful in certain circumstances - for example around deep fakes or policy awareness for proposed legislation, but these tactics would be determined by the institute itself.
To establish and operate under Wise AI philosophical principles, industry-wide best practices should be noted. These principles should be integrated with current standards such as Asilomar Principles, the EU Ethics Guidelines for Trustworthy AI, but these guidelines would have more formalized philosophical reasoning.
Since Isaac Asimov’s fictional Three Laws of Robotics published in Runaround (1942), there have been many instantiations of AI development principles, but the two that most carefully consider philosophical arguments have been the 2016 Asilomar Principles and the 2017 IEEE Ethically Aligned Designed (EAD). Both of these sets of principles were developed by diverse groups of industry, neoacademic, and academic minds across multiple disciplines. These principles were by no means to be a complete and final guidance on how to develop AI, but a starting point.
Broadening the discussion of and operationalizing EAD Recommendations would be a vital goal for a Wise AI organization. There are many more recommendations relevant to sustainable development, personal data, policy, etc., but establishing actionable goals for existing recommendations and identifying gaps is a potential first step. Each EAD issue was developed through a standardized, but open and collaborative consensus-building approach. One such recommendation is that an independent, internationally coordinated body (akin to ISO) should be formed to determine whether AI products are “designed, developed and deployed” ethically. A Wise AI Institute could house and develop functions of such a body, in addition to developing funding models and institutional incentives for interdisciplinary projects also recommended in the EAD document.
Such projects could include a pilot educational programs similar to Scientific Educational Programs in MSL role, where a round-table style program is conducted on a regional basis to disseminate findings, or new guideline/regulatory roll-outs. For example, Bryan Christian commented that Paul Christiano and Leike Andre’s motor simulations at Deep Mind produced a “Platonic ideal” of a backflip for each user that reinforced learning of an agent. They conducted a set of reinforcement learning tasks where the reward function was not observed, but the agent learned about the goal by asking a human to improve on a simulated robot performing a backflip. I could imagine hosting a round-table case study discussion to integrate the ideas of philosophy, and machine learning to probe specific questions from a Wise AI directive. In this example, OpenAI and Google might fund a small dinner discussion where they invitee key academics, customers, and other thought leaders who were up and coming in the field to present and discuss certain technical and philosophical aspects of this study (and others) with the goals of considering the philosophical implications of RHLF and bringing awareness to pipeline activities and upcoming products. These types of programs in the medical industry and quite common, and often provide positive outcomes for the businesses sponsoring the program, the physicians attending, and ultimately the patients who benefit from discussion and networking of their providers.
The EAD principles acknowledge the difficulty of establishing universal ethical principles, but overall they take a pluralist stance asserting that “affective systems to respect the values of the cultures within which they are embedded”. To what extent social media, generative assistants, games, AI companions, or educational tools are “affective” is an important discussion to be held, and are in forums like the 12th International Conference on Affective Computing - however there is a missed opportunity in disseminating these discussions to and furthering collaboration with appropriate stakeholders at relevant companies, adjacent organizations, and consumers. A TTL function would start to bridge this gap, bringing such findings to various customers and stakeholders through their field interactions.
Based on classical ethics, the EAD suggests that AI developers should prioritize learning about well-being concepts, but there is very little discussion of the ontology or phenomenology of well-being in the community. Historically, rationalist communities have ventured into the immaterial; the general treatment of affect as “irrational” and lack of interest in Richard Ngo’s concept of meta-rationality obviates this. Operating with wise AI principles would require deeper interrogation into immateriality that is a component of the Lucretius problem, but this must be done in a way that suits all parties and faces inherent incommensurability. For more on this see Greenaway’s discussion of scale-relative ontology.
 | A guest post by
|