Aneesh Sathe

Author: Aneesh Sathe

  • Briefing: The State of Explainable AI (XAI) and its Impact on Human-AI Decision-Making

    This post is a sloptraption, my silk thread in the CloisterWeb. The post was made with the help of NotebookLM. You can chat with the essay and the sources here: XAI NotebookLM Chat

    I. Executive Summary

    The field of Explainable AI (XAI) aims to make AI systems more transparent and understandable, fostering trust and enabling informed human-AI collaboration, particularly in high-stakes decision-making. Despite significant research efforts, XAI faces fundamental challenges, including a lack of standardized definitions and evaluation frameworks, and a tendency to prioritize technical “faithfulness” over practical utility for end-users. A new paradigm emphasizes designing explanations as a “means to an end,” grounded in statistical decision theory, to improve concrete decision tasks. This shift necessitates a human-centered approach, integrating human factors engineering to address user cognitive abilities, potential pitfalls, and the complexities of human-AI interaction. Practical challenges persist in implementation, including compatibility, integration, performance, and, crucially, inconsistencies (disagreements) among XAI methods, which significantly undermine user trust and adoption.

    Poppies and Daisies (1867) by Odilon Redon. Original from the Art Institute of Chicago. Digitally enhanced by rawpixel.

    II. Core Concepts and Definitions

    • Explainable AI (XAI): A research area focused on making AI system behaviors and decisions understandable to humans, aiming to increase trustworthiness, transparency, and usability. The term itself gained prominence around 2016, though the need for explainability in AI has existed for decades.
    • Contextual Importance and Utility (CIU): A model-agnostic, universal foundation for XAI based on Decision Theory. CIU extends the traditional linear notions of “importance” (of an input) and “utility” (of an input value toward an outcome) to non-linear AI models. It explicitly quantifies how the importance of an input and the utility of its values change based on other input values (the “context”).
    • Contextual Importance (CI): Measures how much modifying a given set of inputs in a specific context affects the output value.
    • Contextual Utility (CU): Quantifies how favorable (or unfavorable) a particular input value is for the output in a given context, relative to the minimal and maximal possible output values.
    • Distinction from Additive Feature Attribution Methods (e.g., LIME, SHAP): CIU is theoretically more sound for non-linear models as it considers the full range of input variations, not just local linearity (partial derivatives). Additive methods lack a “utility” concept and might produce misleading “importance” scores in non-linear contexts.
    • Decision Theory: “A branch of statistical theory concerned with quantifying the process of making choices between alternatives.” It provides clear definitions of input importance and utility, intended to support human decision-making.
    • Human Factors Engineering (HFE): An interdisciplinary field focused on optimizing human-system interactions by understanding human capabilities and limitations. It aims to design systems that enhance usability, safety, and efficiency, and is crucial for creating human-centered AI.
    • Key HFE Principles: User-Centered Design, Minimizing Cognitive Load, Consistency and Predictability, Accessibility and Inclusivity, Error Prevention and Recovery, Psychosocial Considerations, Simplicity and Clarity, Flexibility and Efficiency, and Feedback.
    • Explainability Pitfalls (EPs): Unanticipated negative downstream effects from adding AI explanations that occur without the intention to manipulate users. Examples include misplaced trust, over-estimating AI capabilities, or over-reliance on certain explanation forms (e.g., unwarranted faith in numerical explanations due to cognitive heuristics). EPs differ from “dark patterns,” which are intentionally deceptive.
    • Responsible AI (RAI): A human-centered approach to AI that “ensures users’ trust through ethical ways of decision making.” It encompasses several core pillars:
    • Ethics: Fairness (non-biased, non-discriminating), Accountability (justifying decisions), Sustainability, and Compliance with laws and norms.
    • Explainability: Ensuring automated decisions are understandable, tailored to user needs, and presented clearly (e.g., through intuitive UIs).
    • Privacy-Preserving & Secure AI: Protecting data from malicious threats and ensuring responsible handling, processing, storage, and usage of personal information (security is a prerequisite for privacy).
    • Trustworthiness: An outcome of responsible AI, ensuring the system behaves as expected and can be relied upon, built through transparent, understandable, and reliable processes.

    III. Main Themes and Important Ideas

    A. The Evolution and Current Shortcomings of XAI Research

    • Historical Context: The need for explainability in AI is not new, dating back to systems like MYCIN in 1975, which struggled to explain numerical model reasoning. Early efforts focused on “intrinsic interpretability” or “interpretable model extraction” (extracting rules from models), while “post-hoc interpretability” (explaining after the fact) was proposed as early as 1995 but initially neglected.
    • Modern Re-emergence and Limitations: The term “Explainable AI (XAI)” was popularized around 2016, but current research often “tends to ignore existing knowledge and wisdom gathered over decades or even centuries by other relevant domains.” Most XAI work relies on “researchers’ intuition of what constitutes a ‘good’ explanation, while ignoring the vast and valuable bodies of research in philosophy, psychology, and cognitive science of how people define, generate, select, evaluate, and present explanations.”
    • Focus on Technical Metrics over User Utility: Many XAI papers prioritize “internal validity like deriving guarantees on ‘faithfulness’ of the explanation to the model’s underlying mechanisms,” rather than focusing on how explanations improve human task performance. This can lead to methods that are “non-robust or otherwise misleading.”
    • The “Disagreement Problem”: A significant practical challenge where different XAI methods (e.g., SHAP, LIME) generate “conflicting explanations that lead to feature attributions and interpretability inconsistencies,” making it difficult for developers to trust any single explanation. This is reported as the most severe challenge by practitioners, despite being less frequently reported as an initial technical barrier.

    B. The “Means to an End” Paradigm for XAI

    • Explanations as Decision Support: A core argument is that “explanations should be designed and evaluated with a specific end in mind.” Their value is measured by the “expected improvement in performance on the associated task.”
    • Formalizing Use Cases as Decision Problems: This framework suggests representing tasks as “decision problems,” characterized by actions under uncertainty about the state of the world, with a utility function scoring action-state pairs. This forces specificity in claims about explanation effects.
    • Value of Information: Explanations are valuable if they convey information about the true state to the agent, either directly (e.g., providing posterior probability) or indirectly (helping the human better integrate existing information into their decision).
    1. Three Definitions of Explanation Value:Theoretic Value of Explanation (∆E): The maximum possible performance improvement an idealized, rational agent could gain from accessing all instance-level features (over no information). This acts as a sanity check: if this value is low, the explanation is unlikely to help boundedly rational humans much.
    2. Potential Human-Complementary Value of Explanation (∆Ecompl): The potential improvement the rational agent could gain from features beyond what’s already contained in human judgments.
    3. Behavioral Value of Explanation (∆Ebehavioral): The actual observed improvement in human decision performance when given access to the explanation, compared to not having it (measured via randomized controlled experiments).
    • Critique of Idealized Agent Assumption: While explanations offer no additional value to an idealized Bayesian rational agent (as they are a “garbling” of existing information), they are crucial for imperfect human agents who face cognitive costs or may be misinformed or misoptimizing.

    C. The Critical Role of Human Factors and Human-Centered AI

    • Bridging Algorithmic Complexity and Human Understanding: HFE is essential to “bridge algorithmic complexity with actionable understanding” by ensuring AI systems align with human cognitive abilities and behavioral patterns.
    • Addressing Unintentional Negative Effects (EPs): HFE provides strategies to anticipate and mitigate EPs, such as designing for “user reflection (as opposed to acceptance)” by promoting “mindful and deliberative (system 2) thinking.”
    • Case Study (Numerical Explanations): A study revealed that both AI experts and non-experts exhibited “unwarranted faith in numbers” (numerical Q values for robot actions), perceiving them as signaling intelligence and potential actionability, even when their meaning was unclear. This demonstrates an EP where well-intentioned numerical transparency led to misplaced trust.
    • Seamful Design: A proposed HFE design philosophy that “strategically reveal relevant information that augments system understanding and conceal information that distracts.” This promotes reflective thinking by introducing “useful cognitive friction,” for example, through interactive counterfactual explanations (“what-if” scenarios).
    • Iterative Design and Stakeholder Engagement: Addressing EPs requires an “iterative approach that allows insights from evaluation to feedback to design,” involving “users as active partners” through participatory design methods.
    • Reframing AI Adoption: HFE advocates for a mindset shift from uncritical “acceptance-driven AI adoption” to “critical reflection,” ensuring AI is “worthy of our trust” and that users are aware of its capabilities and limitations. This resists the “move fast and break things” mentality.
    • Human-AI Relationship in Decision-Making: For high-stakes decisions, AI systems should be seen as “empowerment tools” where the human decision-maker retains responsibility and needs to “justify their decision to others.” XAI is key to making the AI’s role clear and building trust.
    • “Justification” vs. “Explanation”: Some differentiate explanation (understanding AI’s intrinsic processes) from justification (extrinsic information to support AI’s results, e.g., patient history, contrastive examples). Both are crucial for human decision-makers.
    • Mental Models: Effective human-AI collaboration relies on humans developing appropriate mental models of the AI system’s capabilities and limitations. XAI should facilitate this “human-AI onboarding process.”

    D. Practical Challenges in XAI Adoption and Solutions

    1. Catalog of Challenges (from Stack Overflow analysis):Model Integration Issues (31.07% prevalence): Difficulty embedding XAI techniques into ML pipelines, especially with complex models.
    2. Visualization and Plotting Issues (30.01% prevalence): Problems with clarity, interpretability, and consistency of visual XAI outputs.
    3. Compatibility Issues (20.36% prevalence): XAI techniques failing across different ML frameworks or hardware due to mismatches.
    4. Installation and Package Dependency Issues (8.14% prevalence): Difficulties in setting up XAI tools due to conflicts or poor documentation.
    5. Performance and Resource Issues (6.78% prevalence): High computational costs and memory consumption.
    6. Disagreement Issues (2.11% prevalence, but most severe): Conflicting explanations from different XAI methods.
    7. Data Transformation/Integration Issues (1.50% prevalence): Challenges in formatting or merging data for XAI models.
    • Perceived Severity vs. Prevalence: While Model Integration and Visualization/Plotting are most prevalent as technical hurdles, Disagreement Issues are perceived as the most severe by practitioners (36.54% rank highest), as they undermine trust and effective decision-making once tools are implemented.
    • Recommendations for Improvement: Practitioners prioritize:
    • Better Documentation and Tutorials (55.77% strongly agree): Clear, structured guides.
    • Clearer Guidance on Best Practices (48.07% strongly agree): Standardized methodologies.
    • Simplified Configuration and Setup (40.38% strongly agree): Easier onboarding.
    • User-Friendly Interfaces and Improved Visualization Tools: More intuitive and interactive tools.
    • Enhanced Integration with Popular ML Frameworks and Performance Optimization.
    • Addressing Disagreement and Consistency: Acknowledge disagreements and guide users in selecting reliable explanations.

    IV. Gaps and Future Directions

    • Lack of Standardization: XAI still lacks standardized definitions, metrics, and evaluation frameworks, hindering consistent assessment and comparison of methods.
    • Limited Empirical Validation: More situated and empirically diverse human-centered research is needed to understand stakeholder needs, how different user characteristics (e.g., expertise, background) impact susceptibility to EPs, and how explanations are appropriated in unexpected ways.
    • Beyond “Accuracy”: Future research should go beyond basic performance metrics to holistically evaluate human-AI relationships, including reliance calibration, trust, and understandability.
    • Taxonomy of EPs: Developing a taxonomy of explainability pitfalls to better diagnose and mitigate their negative effects.
    • Longitudinal Studies: Needed to understand the impact of time and repeated interaction on human-AI decision-making and trust dynamics.
    • Interdisciplinary Collaboration: Continued and enhanced collaboration among HFE, cognitive science, and AI engineering is crucial to develop frameworks that align AI decision-making with human cognitive and operational capabilities, and to address ethical and accountability challenges comprehensively.
    • Benchmarking for Responsible AI: Creation of benchmarks for various responsible AI requirements (ethics, privacy, security, explainability) to quantify their fulfillment.
    • “Human-in-the-loop”: Further development of this concept within responsible AI, emphasizing the human’s role in checking and improving systems throughout the lifecycle.
    • Trade-offs: Acknowledge and manage inherent trade-offs between different responsible AI aspects (e.g., robustness vs. explainability, privacy vs. accuracy).

    V. Conclusion

    The transition of AI from low-stakes to high-stakes domains necessitates a robust and human-centric approach to explainability. Current XAI research must evolve beyond purely technical considerations to embrace principles from Decision Theory and Human Factors Engineering. The development of frameworks like CIU and the rigorous evaluation of explanations as “means to an end” for specific decision tasks are critical steps. Addressing practical challenges identified by practitioners, especially the pervasive “disagreement problem” and the occurrence of “explainability pitfalls,” is paramount. Ultimately, achieving Responsible AI requires a dynamic, interdisciplinary effort that prioritizes human understanding, trust, and ethical considerations throughout the entire AI lifecycle, ensuring AI serves as an effective and accountable partner in human decision-making.

  • AI: Explainable Enough

    They look really juicy, she said. I was sitting in a small room with a faint chemical smell, doing one my first customer interviews. There is a sweet spot between going too deep and asserting a position. Good AI has to be just explainable enough to satisfy the user without overwhelming them with information. Luckily, I wasn’t new to the problem. 

    Nuthatcher atop Persimmons (ca. 1910) by Ohara Koson. Original from The Clark Art Institute. Digitally enhanced by rawpixel.

    Coming from a microscopy and bio background with a strong inclination towards image analysis I had picked up deep learning as a way to be lazy in lab. Why bother figuring out features of interest when you can have a computer do it for you, was my angle. The issue was that in 2015 no biologist would accept any kind of deep learning analysis and definitely not if you couldn’t explain the details. 

    What the domain expert user doesn’t want:
    – How a convolutional neural network works. Confidence scores, loss, AUC, are all meaningless to a biologist and also to a doctor. 

    What the domain expert desires: 
    – Help at the lowest level of detail that they care about. 
    – AI identifies features A, B, C, and that when you see A, B, & C it is likely to be disease X. 

    Most users don’t care how a deep learning really works. So, if you start giving them details like the IoU score of the object detection bounding box or if it was YOLO or R-CNN that you used their eyes will glaze over and you will never get a customer. Draw a bounding box, heat map, or outline, with the predicted label and stop there. It’s also bad to go to the other extreme. If the AI just states the diagnosis for the whole image then the AI might be right, but the user does not get to participate in the process. Not to mention regulatory risk goes way up.

    This applies beyong images, consider LLMs. No one with any expertise likes a black box. Today, why do LLMs generate code instead of directly doing the thing that the programmer is asking them to do? It’s because the programmer wants to ensure that the code “works” and they have the expertise to figure out if and when it goes wrong. It’s the same reason that vibe coding is great for prototyping but not for production and why frequent readers can spot AI patterns, ahem,  easily.  So in a Betty Crocker cake mix kind of way, let the user add the egg. 

    Building explainable-enough AI takes immense effort. It actually is easier to train AI to diagnose the whole image or to give details. Generating high-quality data at that just right level is very difficult and expensive. However, do it right and the effort pays off. The outcome is an AI-Human causal prediction machine. Where the causes, i.e. the median level features, inform the user and build confidence towards the final outcome. The deep learning part is still a black box but the user doesn’t mind because you aid their thinking. 

    I’m excited by some new developments like REX which sort of retro-fit causality onto usual deep learning models. With improvements in performance user preferences for detail may change, but I suspect that need for AI to be explainable enough will remain. Perhaps we will even have custom labels like ‘juicy’.

    Fediverse Reactions

  • My Road to Bayesian Stats

    By 2015, I had heard of Bayesian Stats but didn’t bother to go deeper into it. After all, significance stars, and p-values worked fine. I started to explore Bayesian Statistics when considering small sample sizes in biological experiments. How much can you say when you are comparing means of 6 or even 60 observations? This is the nature work at the edge of knowledge. Not knowing what to expect is normal. Multiple possible routes to a seen a result is normal. Not knowing how to pick the route to the observed result is also normal. Yet, our statistics fails to capture this reality and the associated uncertainties. There must be a way I thought. 

    Free Curve to the Point: Accompanying Sound of Geometric Curves (1925) print in high resolution by Wassily Kandinsky. Original from The MET Museum. Digitally enhanced by rawpixel.

    I started by searching for ways to overcome small sample sizes. There are minimum sample sizes recommended for t-tests. Thirty is an often quoted number with qualifiers. Bayesian stats does not have a minimum sample size. This had me intrigued. Surely, this can’t be a thing. But it is. Bayesian stats creates a mathematical model using your observations and then samples from that model to make comparisons. If you have any exposure to AI, you can think of this a bit like training an AI model. Of course the more data you have the better the model can be. But even with a little data we can make progress. 

    How do you say, there is something happening and it’s interesting, but we are only x% sure. Frequentist stats have no way through. All I knew was to apply the t-test and if there are “***” in the plot, I’m golden. That isn’t accurate though. Low p-values indicate the strength of evidence against the null hypothesis. Let’s take a minute to unpack that. The null hypothesis is that nothing is happening. If you have a control set and do a treatment on the other set, the null hypothesis says that there is no difference. So, a low p-value says that it is unlikely that the null hypothesis is true. But that does not imply that the alternative hypothesis is true. What’s worse is that there is no way for us to say that the control and experiment have no difference. We can’t accept the null hypothesis using p-values either. 

    Guess what? Bayes stats can do all those things. It can measure differences, accept and reject both  null and alternative hypotheses, even communicate how uncertain we are (more on this later). All without making assumptions about our data.

    It’s often overlooked, but frequentist analysis also requires the data to have certain properties like normality and equal variance. Biological processes have complex behavior and, unless observed, assuming normality and equal variance is perilous. The danger only goes up with small sample sizes. Again, Bayes requires you to make no assumptions about your data. Whatever shape the distribution is, so called outliers and all, it all goes into the model. Small sample sets do produce weaker fits, but this is kept transparent. 

    Transparency is one of the key strengths of Bayesian stats. It requires you to work a little bit harder on two fronts though. First you have to think about your data generating process (DGP). This means how do the data points you observe came to be. As we said, the process is often unknown. We have at best some guesses of how this could happen. Thankfully, we have a nice way to represent this. DAGs, directed acyclic graphs, are a fancy name for a simple diagram showing what affects what. Most of the time we are trying to discover the DAG, ie the pathway of a biological outcome. Even if you don’t do Bayesian stats, using DAGs to lay out your thoughts is a great. In Bayesian stats the DAGs can be used to test if your model fits the data we observe. If the DAG captures the data generating process the fit is good, and not if it doesn’t. 

    The other hard bit is doing analysis and communicating the results. Bayesian stats forces you to be verbose about your assumptions in your model. This part is almost magicked away in t-tests. Frequentist stats also makes assumptions about the model that your data is assumed to follow. It all happens so quickly that there isn’t even a second to think about it. You put in your data, click t-test and woosh! You see stars. In Bayesian stats stating the assumptions you make in your model (using DAGs and hypothesis about DGPs) communicates to the world what and why you think this phenomenon occurs. 

    Discovering causality is the whole reason for doing science. Knowing the causality allows us to intervene in the forms of treatments and drugs. But if my tools don’t allow me to be transparent and worse if they block people from correcting me, why bother?

    Richard McElreath says it best:

    There is no method for making causal models other than science. There is no method to science other than honest anarchy.

    Fediverse Reactions

  • The Keel

    Tonight we cast our nets
    In foreign waters
    Now we are new
    Tomorrow we’ll belong
    Then the sea'll tug and pull
    Time to be gone

    So, let’s
    Kiss the nymphs
    Meet the crabs
    In their bucket games
    Feel their stabs

    Our plans are small
    But they are spread
    A thousand ports
    Before we are dead

    Image: Sailing off Gloucester (ca.1880) by Winslow Homer.

  • The secret flag of content

    I don’t have any fun when I use LLMs to write. It may have perceived utility: popping out a LinkedIn article or two everyday. But I bet no one is actually reading. It’s a strip mall for a thumb stroll.  

    LLMs suck at writing. The summaries that LLMs give with the “Deep Research” are so poor in quality that I start to skim it. Yes, I skim the thing that is already a summary. 

    Let’s do a small experiment using the  Harry G. Frankfurt Essay On Bullshit: 

    Here are HGF’s opening lines: 

    “One of the most salient features of our culture is that there is so much bullshit. Everyone knows this. Each of us contributes his share.”

    Now here is ChatGPT o3, with the advantage of having HGF in the training data and extra contextualizing from me “theory of bullshit”: 

    Prompt: write me an introductory paragraph on the theory of bullshit 

     Result: “Bullshit isn’t just lying gone sloppy; it’s a distinct communicative strategy built on indifference to truth. Where the liar knows the facts and works to conceal or invert them, the bullshitter treats truth-value as irrelevant raw material—selecting, mixing, or inventing claims to achieve an effect (status, alignment, vibe) rather than to inform. The result is epistemic pollution: signals that…”

    Look! It’s the secret flag of “content”:  “Blah isn’t just X; Blah is Y”.

    AI generated flag. As dead as the text on it is wrong.

    I cannot assimilate this, it definitely can’t help me synthesize anything. There is always better human writing even if it is poorly written. It has a certain aliveness, and that aliveness is contagious enthusiasm. Vibe coding I can understand, you can see bad output in the final result. Vibe coding still manages to change something in my mind: knowing what I don’t want. 

    I don’t mind people using AI at all, I use it alllll the time. Writing with LLMs is no just fun. All this prompting and almost nothing changes in my mind. When an AI rearranges your thoughts it does not rearrange your brain. 

    Fediverse Reactions

  • What do platforms really do? 

    In 1986, David S. Landes wrote the essay, ‘What Do Bosses Really Do?’. He argues that the historical role of the ‘boss’ was an essential function for organizing production and connecting producers to markets. Digital platforms have become the new bosses. Platforms have the same functions of market creation, labor specialization, and management, but they have replaced the physical factory floor with algorithmic management. While their methods are novel, platforms are the direct descendants of the merchant-entrepreneurs and factory owners Landes described, solving the same historical problems of production in remarkably similar ways.

    Design for a Teacup (1880-1910) painting in high resolution by Noritake Factory. Original from The Smithsonian Institution. Digitally enhanced by rawpixel.

    So, why am I posting this on my own blog and not on a “platform”? I don’t view writing as a financial transaction. It is a hobby. By putting the financialization lens front and center, platforms are killing the mental space for hobbies. When you monetize tweets, you create incentive to craft tweets that create engagement in particular ways. Usually not healthy ways. 

    If we think of old media or traditional manufacturing, we can compare them to guilds. Guilds kept up prices and controlled production. With the simplification of tasks factories could hire workers who weren’t as highly skilled but didn’t need to be. Nowadays, why should any newspaper or TV channel’s output be limited by the amount of airtime or page space they have?

    Platforms take unskilled and train them. We are in the age of specialization of ideas.  Akin to the “the advantage of disaggregating a productive process”  Platforms leverage this by having many producers explore the same space through millions of different angles. This allows the platform to “purchase exactly that precise quantity of [skill] which is necessary for each process” —paying a viral star a lot and a niche creator a little, perfectly matching reward to market impact. Which is to say platforms make money through whatever sticks.  

    In Landes’s essay, Management became specialized, today management will become algorithmized. Platforms abstract away the issues that factory owners had such as embezzlement of resources, slacking off etc. Platforms don’t care how much or how little you produce, or even if you produce. If you do, the cash is yours (after a cut of course). 

    This may lead to a visceral reaction against platforms. This week when Substack raised a substantial amount they called the writers “the heroes of culture”. This should ring at least a tiny alarm in your head. The platform’s rhetoric of the creator-as-hero is a shrewd economic arrangement. In the putting-out system, the merchant-manufacturer “was able to shift capital expenditures (plant and equipment) to the worker”. Platforms do the same with creative risk. The writer, artist, or creator invests all the time and labor—the “capital” of creation—upfront. If they fail, they bear the entire loss. The platform, like the putter-outer, only participates in the upside, taking its cut from the successful ‘heroes’ while remaining insulated from the failures of the many.

    So what do platforms really do? They have resurrected the essential role of the boss for the digital age. They are the merchant-manufacturers who build the roads to market, and they are the factory owners who discipline production—not with overseers, but with incentive algorithms. By casting the creator as the hero, they obscure their own power and shift the immense risks of creative work onto the individual. While appearing to be mere background IT admins, they are, in fact, the central organizers of production, demonstrating that even in the 21st century, the fundamental challenges of coordinating labor and capital persist, and solving them remains, as it was in the 18th century, a very lucrative role.

    What Do Bosses Really Do?, David S. Landes, The Journal of Economic History, Vol. 46, No. 3 (Sep., 1986), pp. 585-623 (39 pages). https://www.jstor.org/stable/2121476

    Fediverse Reactions

  • Hack, Hacky, Hacker

    A few days ago I wrote about the beauty of great documentation; this is the evil twin post.

    The spectrum of meaning across the words hack, hacky, and hacker form a horseshoe when thinking about postures toward life. On either ends are the most difficult options. Being either a hack or a hacker requires dedication and both approaches narrow your world. Being hacky, taking imperfect shortcuts, in the world is immensely satisfying. It is play disguised as problem solving. 

    Fox by Arnold Peter Weisz Kubincan. Original public domain image from Web umenia

    A successful hack takes tremendous effort and dedication just to pretend to be great at something. Humans are great at spotting and discarding hacks. It takes a true master to fool a large enough population and build financial columns under the smoke. Being a hack is constant desperation, there is no play. It is no way to live. 

    On the other end of the same horseshoe as the hack, is hacking. Here, you are actually achieving something difficult enough to require mastery. “Playfully doing something difficult, whether useful or not, that is hacking.” says Richard Stallman. Now, I’m all for the playful, the difficult, and the useful, but not the “or not”. At minimum hacking should be in service of a prank. Doing things just because is like felling a tree in a forest when no one is around. At least a jump scare is a sine qua non (the dictionary is working :P). 

    Most systems, especially computers are designed by people for people like you and me who are neither very bright nor very invested in the thing. We want to not have the problem. You can always walk away but that is neither fun, nor useful, and certainly not hard. My favored way is to take the Nakatomi Tunnel through problems. Be hacky. Try enough approaches, push buttons that may do the thing you want until the alignment is just so and you slip through. Effectiveness here = solving many real-world problems quickly while preserving playful momentum.

    I want to draw a distinction here from the oversubscribed idea of jugaad. Jugaad was once framed as creative improvisation. It is not. I do not care for jugaad. To make something substandard and expect people to accept it is no way to be in the world. Build good stuff, be hacky route through the small issues.

    A hacky mindset is a foxy mindset and not just in the Hendrix way. The Hedgehog and the Fox is a great essay by Isaiah Berlin where he talks about the two kinds of people in the world. Hedgehogs, are great at one big thing. Foxes are mediocre at many things. Foxes thrive on lateral moves and opportunistic shortcuts, you know, hackiness. The hacky, foxy approach to life is more my style. 

    Breadth, speed, and joy beat fakery and fixation every time

    Fediverse Reactions

  • A Good Dictionary

    Yesterday I wrote about good documentation opening doors to options you didn’t realize you had. In the book On Writing Well Zinsser mentions how one of his key tools is the dictionary. That got me curious about the limitations about the dictionaries available to us. This is not just about the dictionary on the bookshelf but the ones that we have in-context access to. The ones on our computer and phones. 

    In my searches I came across this post by James Somers who references another great writer John McPhee and his article Draft No. 4. McPhee shows us how the dictionary is to be used. The crux is that modern dictionaries have taken all the fun out and left all the crud in. The old way is the proper way to play with words. 

    J.S ends with instructions on how to install the (apparently perfect) 1913 version of Webster’s dictionary. Unfortunately, his instructions are a little out of date. Which is to be accepted since he’s talking to people 10 years in his future. Luckily for us Corey Ward from speaking to use from just 5 years ago had updated instructions for MacOS that mostly still work.

    I’m updating Corey’s instructions below:

    1. Get the latest release for Webster’s 1913 from the Github Releases page for WebsterParser. Download the file: websters-1913.dictionary.zip and unzip it. You will see a folder like file with the extension .dictionary.
    2. Open the Dictionary app on your computer, and select File > Open Dictionaries Folder from the menu, or navigate manually to ~/Library/Dictionaries.
    3. Unzip the file, and move the resulting websters-1913.dictionary file into the dictionaries folder that you opened.
    4. Restart the Dictionary app if it is open (important), then open Dictionary>Settings (⌘,). At the bottom of the list of dictionaries you should see Webster's Unabridged Dictionary (1913) in the list. Check the box, and optionally drag it up in the list to the order you’d like.

    The dictionary is also available online if you don’t want to install.

    The best option is probably the OED . It’s expensive, but you may get access through your library. 

    Wordnik also cool. 

    Through J.S. I also discovered this interesting site: Language Log. They get really deep into language. I mean how much can you write about Spinach, apparently a lot. 

    I’d love to get back to a world where the internet was used in its raw form. If you are reading my posts, please do comment, share your site/blog and your posts. Social media is also good. More from Somers.

  • Divine Documentation

    Dad was about my age when he said that reading the manual was better than hypothesis driven button pressing. For teenage me, that took too long. Sure, I may have crashed a computer or two but following my gut got me there. Of course my gut isn’t that smart. In the decades preceding, devices had converged on a common pattern language of buttons. Once learned, the standard grammar of action would reliably deliver me to my destination. 

    Image of a nebula taken by the Hubble Telescope.

    In programming I was similarly aided by the shared patterns across MATLAB, Python, R, Java, Julia, and even HTML. In the end however, dad was right. Reading documentation is the way. Besides showing correct usage, manuals create a new understanding of my problems. I am able to play with tech thanks to the people that took the effort and the care to create good documentation. This is not limited to code and AI. During the startup years, great handbooks clarified accounting, fundraising, and regulations, areas foreign to me.

    I love good documentation and I write documentation. Writing good documentation is hard. It is an exercise in deep empathy with my user. Reaching into the future to give them all they need is part of creating good technology. Often the future user is me and I like it when past me is nice to now me. If an expert Socratic interlocutor is like weight training, documentation is a kindly spirit ancestor parting the mist. 

    Maybe it’s something about being this age but now I try to impart good documentation practices to my teams. I also do not discourage pressing buttons to see what happens. Inefficient, but discovery is a fun way to spike interest.

    Meanwhile, I’m reading a more basic kind of documentation. Writing English. Having resolved to write more, I’m discovering that words are buttons. Poking them gets me to where I want, but not always. Despite writerly ambitions, the basics are lacking. This became apparent recently when I picked up the book Artful Sentences by Virginia Tufte*. It’s two hundred and seventy pages of wonderful sentences dissected to show their mechanics. I was lost by page 5. The book is, temporarily, in my anti-library. 

    So, I’m going to the basics, Strunk and White, and William Zinsser. I’m hoping that Writing to Learn (finished) and On Writing Well (in progress) provide sufficient context about reasons to write to make the most of S&W, for the how, then somewhere down the road, savor Tufte. 

    * Those dastardly Tuftes are always making me learn some kind of grammar.

    Fediverse Reactions

  • The Plato Plateau

    This post started off as a joke. I was attempting to snow clone the Peter Principle for philosophy. It led to a longer thread of thoughts. But first, the snow clone: 

    The Plato Plateau: People philosophize to the level of their anxiety.

    Smoking farmer with branches by Kono Bairei (1844-1895). Digitally enhanced from our own original 1913 edition of Barei Gakan.
    1. Anxiety is the realization that you have absolute choice over life – Kierkegaard. Anxiety, in this context is not nervousness. It is a positive thing when harnesses. We harness it everyday.  
    2. Anxiety is a generative. Anxiety creates identity by locating stable places to launch exploration.
    3. Action, exploration, and anxiety are a motor. Anxiety → exploration → action → refreshed identity. Inaction leads to identity death
    4. Realizing you are radically free to choose can also lead to a forest of perceived signals. These can be an overwhelming inbox or simply overloaded ambition.
    5. When anxiety overwhelms it becomes difficult to tell signal from noise.
    6. Tools like GTD crash anxiety. When overwhelmed, GTD works well. When there is too little anxiety identity becomes ephemeral. 
    7. GTD isn’t a means to nirvana: GTD integrates 10k, 30k foot views to reintroduce future anxiety.
    8. When your identity is smeared across too many anxieties you declare anxiety bankruptcy and crash your identity in some safe spot. Journals, sabbaticals, quitting.
    9. Like the parable of the rock soup, vaporized anxiety needs a place to condense onto. Ideally something disposable but sufficient to let your identity create an “ordered world of meaning”
    10. Life examination occurs with identity crashes. Philosophy provides just enough of a toehold in the abstract to spur action in the actual. 
    11. Philosophy is a way to spur action absent anxiety/identity. We pick the philosophy depending on the degree of identity loss.
    12. Philosophy can be broadly sorted as:
      1. Survival – laws and tactics oriented
      2. Social Cohesion- harmony, virtue ethics, etiquette 
      3. Systems level order – algorithms and protocols oriented
      4. Self Knowledge and Meaning – reflecting on existing and consciousness 
      5. Meta-systems – theorizes about theories
    13. Most scientists and builders work best at level 3 systems level order. Going lower, i-ii, for environmental crises and higher, iv-v, for internal crises. 
    14. Complexity of selected philosophy is not superiority. A rung’s usefulness matches your identity state and environment, not some civilizational high score.
    15. Philosophy as Periodic Maintenance: Crashing and philosophy sampling are maintenance actions on the place called identity.
    Fediverse Reactions