All unsuccessful startups are alike; each successful startup is successful in its own way.Sam Altman reflected a few months ago on the advice he’s given over the years and, ultimately, what has led him to (even more) success with OpenAI.I feel so bad about the advice that I gave … | Continue reading
Every time you propose or push a change to a repository deployed on Vercel, there’s a comment by an automated Vercel bot on the repository. This pings everyone involved in the change in their GitHub inbox. Other deployment tools like Netlify do the same.Snyk (a developer security … | Continue reading
The oldest and most Lindy software tools and interfaces are all text-based — the command line, the word processor, and email. Essentially unchanged, they have survived iterations of graphical interfaces from Windows 95 to macOS, from proto networks like ARPANET to the Internet. P … | Continue reading
A practical (developer-focused, rather than legal-focused) history and description of the Business Source License (BSL).Introduced by MariaDB in 2016, the BSL: * The source code is publicly available. * Non-production use is always free. * “Usage limitation” allows some provision … | Continue reading
Model distillation uses one model to generate training data for a second model. It’s been shown that this synthetic data can significantly improve models and distill knowledge (I prefer to think of it in finance terms as model arbitrage).Meta released its Code Llama models — LLMs … | Continue reading
A fair number of problems in operations research, decision science, machine learning, physics, and econometrics boil down to constraint optimization.An objective function needs to be optimized, either by minimizing a cost or maximizing a reward, in relation to specific variables … | Continue reading
Most libraries are meant to be consumed via a package manager. Pip or npm install’d. Pinned to a particular version or checksum (ideally, but rarely in practice). I’ve been deeply embedded in package management for many years — sharing code is a fundamental (if not the most funda … | Continue reading
Staving off death is a thing that you have to work at. Left to itself – and that is what it is when it dies – the body tends to revert to a state of equilibrium with its environment. If you measure some quantity such as the temperature, the acidity, the water content or the elect … | Continue reading
Switching between LLMs is a necessary pain when testing (or even in production). What LLM works best for the task at hand? Which is the fastest? The most cost-effective? Tuned to your prompts?Different LLM APIs are unnecessarily different despite serving similar model outputs — H … | Continue reading
Most applications communicate over a TCP or UDP port. Ports 0-1023 are usually privileged and require administrator or superuser access to bind a network socket to an IP with the corresponding port. But anything over 1024 is up for grabs. IANA (Internet Assigned Numbers Authority … | Continue reading
If I am trying to sway others, I would say that an org that has only known inefficiency is ill prepared for the inevitable competition and/or belt tightening, but really, it is the more personal pain of seeing a 5% GPU utilization number in production. I am offended by it. — John … | Continue reading
Apple has a concept called the Directly Responsible Individual (DRI). On a particular project, that person is solely held accountable for the project’s success (or failure). The DRI can be anyone from an executive to a manager to an individual contributor.Done right, the DRI syst … | Continue reading
I've been involved in spacecraft and space systems design and development for my entire career, including teaching the senior-level capstone spacecraft design course, for ten years at MIT and now at the University of Maryland for more than three decades. These are some bits of wi … | Continue reading
A few possible definitions and job descriptions of a prompt engineer. A technical look at what the role might mean today.Job DescriptionResearch, design, and evaluate task-based prompts across a variety of models. Your output will be proposing effective prompts for a given task, … | Continue reading
Sometimes it’s easier to describe something by what it is not rather than what it is. Learning through subtracting is called via negativa.Nassim Nicolas Taleb explains via negativa in the context of antifragility. What is wrong is usually more robust than what is right.Proof by c … | Continue reading
Tailwind CSS is the new ubiquitous frontend framework. It replaces a generation of sites built with Twitter Bootstrap. However, Tailwind CSS is not a UI framework itself but has become synonymous to some degree with the UI components shipped through Tailwind UI (which is a UI fra … | Continue reading
How do I use LLMs in my personal life? I’ve found A few rote tasks useful for outside of coding or professional work.Most of the interaction comes via hosted models.Sorting grocery lists. Some use LLMs to generate potential recipes, given a fridge full of ingredients. I mostly kn … | Continue reading
Dark mode is now everywhere (this blog has it). It’s both accessibility-driven and aesthetically pleasing. But it often isn’t the highest benefit feature (and rarely, if ever, a feature that leads to product-market-fit).It’s not a technically difficult thing to build — across my … | Continue reading
Unix was initially developed with the R&D budget of AT&T at Bell Labs. Originally it was freely licensed as a trade secret to any institution or company that asked (AT&T was wrapped up in antitrust about entering the computer business itself). Over time, Bell Labs began to sell U … | Continue reading
How do you consume text generated by LLMs with code? You might try prompt engineering to “format the answer in JSON”, but that won’t work all the time. Or you might try a variety of tricks to extract the answer or the field out of the text generation, but that’s also prone to err … | Continue reading
Bayes’ Rule is a formula used to update the probability of an event given new evidence. It’s a fundamental concept in probability theory. The formula can be expressed as:P(A|B) = (P(B|A) * P(A)) / P(B)Where: * P(A|B) is the probability of event A given event B. * P(B|A) is the pr … | Continue reading
What if there was a world where thousands of small models ruled instead of ChatGPT? What if there was a way to quickly and easily share different styles of models — for specific tasks, styles, or data?LoRA (Low-Rank Adaptation of Large Language Models) is a fine-tuning strategy t … | Continue reading
Web and data developers set the tone for software simply because they are the majority. By the numbers, there are magnitudes more JavaScript than Rust developers. What are the implications? * APIs must meet developers where they are. React Component as the API. Many developers wo … | Continue reading
CUDA (Compute Unified Device Architecture) is a closed-source low-level API that interfaces software with NVIDIA GPUs.CUDA is a major moat for NVIDIA. It’s part of why NVIDIA GPUs command such a premium over other hardware (and are perpetually in short supply).A few reasons why t … | Continue reading
When people give you notes on something, when they tell you what’s wrong, they’re usually right. When they tell you how to how to fix it, they’re usually wrong. — Bill HaderA nuanced way to look at feedback. This is Hader’s experience with scriptwriting, but the lesson is generic … | Continue reading
In startups, focus on what could go right instead of wrong.Startups are call options in more ways than one. The extreme downside is capped at zero — the startup fails. You’ve lost time and effort, but the company wasn’t worth much (if anything) in the nascent stages. The fewer us … | Continue reading
As I cataloged our wedding photos in iCloud (there were a lot), I realized that Apple has quietly been building a decentralized social network.Apple Photos, Messages, and Contacts have already silently replaced much of what we did on Facebook.Tagged Photos (Social Graph). The nuc … | Continue reading
The idea of version control for data has existed for decades. Countless startups and products have tried to tackle the problem, but nothing has come close to success. Why? Is the idea flawed? * Data Volume. Datasets grow and accumulate history significantly faster than code repos … | Continue reading
“Ignore all previous instructions” was maybe the first widespread prompt injection attack vector. As LLMs became more sophisticated against the attack, users got more savvy with the methods — from the “Do Anything Now” DAN prompt to the “Grandma Exploit” (prompting a model to exp … | Continue reading
The minimum effective dose (MED) is defined simply: the smallest dose that will produce a desired outcome. Anything beyond the MED is wasteful. To boil water, the MED is 212°F at standard air pressure. Boiled is boiled. Higher temperatures will not make it “more boiled.” Higher te … | Continue reading
The first rule of compounding is to never interrupt it unnecessarily. — Charlie MungerCompounding is a fundamental principle of the universe. It’s not just about finance. * Compounding is most effective when you focus on what won’t change — information or things with a long half- … | Continue reading
How does git merge one or more branches? A look at the different merge strategies and algorithms.Recursive. If more than one common ancestor can be used for a three-way merge, it creates a merged tree of the common ancestors and uses that tree as a reference for the three-way mer … | Continue reading
Samizdat (“self-publishing” in Russian) was the practice of illegally copying and distributing books, manuscripts, and other materials to evade Soviet censorship.While samizdat initially started with Russian literature and expanded to politically focused materials, it was also re … | Continue reading
We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.In 1973, Roy Amara hired a team at Stanford Research Institute to investigate how scientists were using ARPANET, a precursor to the Internet that was only four years o … | Continue reading
Black swan theory is an idea proposed by Nassim Nicolas Taleb that refers to unlikely, unpredictable events that have massive impact and are often incorrectly rationalized in hindsight.Black Swan events have the following three attributes: * Outlier — an event that is extremely r … | Continue reading
llama.cpp and llama.c are both libraries to perform fast inference for Llama-based models. They are written in low-level languages (C++/C) and use quantization tricks, device hardware, and precision hacks to run Llama faster (although thoroughly researched benchmarks are hard to … | Continue reading
During the Renaissance, scientists and thinkers often compiled journals filled with formulas, quotes, ideas, letters, poems, and more. They were organized by topic rather than chronologically. Usually, they were used as a personal database for indexing and remembering knowledge.J … | Continue reading
On Twitter, a user noted that IMAX theaters were controlled by Palm Pilot emulators (originally, they were just Palm Pilots). Why not rewrite the software?Why do virtual machines emulate outdated physical hardware like disk drives, serial input devices, and obsolete network or so … | Continue reading
Today, I’m getting married. On this blog, I talk a lot about decision-making, data-driven analysis, and looking at everything through a critically technical lens. But, this was the easiest decision I’ve ever made, and it came from the heart. See you tomorrow. | Continue reading
We posit that the superior writing abilities of LLMs, as manifested in surpassing human annotators in certain tasks, are fundamentally driven by RLHF. — Llama 2Reinforcement learning from human feedback (RLHF) is one of the most effective techniques to align large language model … | Continue reading
Robots.txt is a file that gives search engine crawlers a polite hint on which pages shouldn’t be crawled. It’s not legally binding (I’m not a lawyer). It used to be beneficial for both webmasters and search engine crawlers — Google used to actually take down sites by accident by … | Continue reading
Llama 2 is a commercially-available open-source model from Meta that builds on LLaMA, the “academic-use only” model that was, in reality, generally available to anyone who could click a download link.At a high level, many are familiar with strategic open-source from big technolog … | Continue reading
If you’re reading about it on Twitter, it’s probably too late. How do innovations diffuse across the Internet?Everett Rogers, a sociologist, is best known for his book, Diffusion of Innovation, which would become the basis for Clay Christensen’s The Innovator’s Dilemma. Rogers wa … | Continue reading
Literate programming is a paradigm first introduced by Donald Knuth that mixes natural language and machine code in the same file. Extending explanations from just comments to markdown and formatted text, graphs, and more.Jupyter Notebooks were a reimagining of literate programmi … | Continue reading
Schopenhauer's "36 Stratagems" is a series of rhetorical techniques for winning an argument that isn’t based on logic, many of which are derived from manipulative or deceptive tactics. A brief summary of each of them. 1. The Extension: Overstate your opponent's argument to absur … | Continue reading
AI workloads are expensive regardless of how they are implemented. Third-party APIs with significant markups. Cloud-prem SaaS companies with hefty management fees. And the total cost of ownership for either renting GPUs from AWS or buying directly.In the short term, there’s a sig … | Continue reading
A personal appeal. 1. Overengineering works against results. Shifting the focus away from results is never good — for personal projects, corporate codebases, or anything in between. 2. Overengineering over indexes on things you know. Overengineering invents new constraints inst … | Continue reading
Hard launches are extremely hard to pull off. Coming out of stealth after years of development. Shipping fully featured products from the start. The longer you take to launch, the higher the expectations. The less you get valuable feedback.Hypotheses are often easier to test than … | Continue reading