Verbatim

People ask a lot of strange things. Here's what a month of real questions taught me.

805 real conversations, 16 countries, and one bug I only found because I sat down and actually read the logs.

Verbatim. Jinnah, Gandhi, Einstein, and Darwin, in their own words, cited.

Claude-e-Azam started as a chatbot that answers as Muhammad Ali Jinnah, grounded in nothing but his own real words, with a citation on every claim. I wrote about building that one a while back. I built out the same engine into Verbatim, which now speaks as four people at once: Jinnah, Mahatma Gandhi, Albert Einstein, and Charles Darwin, each one answering only from what they actually wrote, cited the same way. The name was the whole point from the start, get as close as possible to someone's actual words rather than whatever secondhand version of them people already believe.

What I had not done, until last week, was actually sit down and read what people were asking them.

Not glance at a dashboard. Not check that the count was going up. Read the questions, one after another, the way you'd read someone's diary if the diary belonged to four dead historical figures and several thousand strangers. It turned out to be the most useful hour I have spent on this project since I fixed the citation bug, and also the funniest.

Who actually shows up

805 real exchanges over about a month, across all four. Jinnah takes the overwhelming majority, 665 of them, which makes sense given he's the one Verbatim grew out of and the one people search for by name. Gandhi comes in at 79, Einstein at 33, Darwin at 28. I built the four pipelines assuming they'd get roughly comparable traffic. They did not, and I don't think that says anything about the three getting less traffic so much as it says Jinnah has a head start of an entire earlier product.

The geography surprised me more than the persona split did. Lahore alone accounts for 558 of the 805, which is not surprising once you say it out loud, Jinnah finding his deepest audience in the city he's most associated with. But past that first wall of Lahore, the tail is longer than I expected. Rawalpindi, Karachi, Quetta, Ludhiana, then real single conversations from London, Dubai, Dhaka, Kolkata, Kathmandu, Singapore, Toronto, Melbourne, and a scattering of towns in Germany and the US I had to look up on a map to confirm were real places. Sixteen countries total, from something I built mostly for myself, at three in the morning, because a phrase wouldn't leave my head.

Then I got to the actual questions

Most of what people ask is exactly what you'd hope for. What was the two-nation theory. Why did you break with Congress. What was the Salt March. Did you envision a radical Pakistan. Genuine, curious, sometimes pointed questions from people who clearly wanted a real answer and were willing to read one.

And then there's the other pile.

Without going into the specifics, since these are real messages from real strangers and not really mine to repeat, people ask all sorts of things that have nothing to do with history. Someone asked Jinnah to write them some Python code. He said no, that his work was law and the founding of Pakistan, not programming. That one made me laugh, but it's also a small useful thing to notice: if a chatbot can be talked into doing your homework or writing code for you, that's not a cute feature, it's someone else using it for free compute that has nothing to do with what it's there for. Staying in character here comes from the same place as not making things up. Same rule, just showing up in a different kind of question.

Some people push harder, saying rude or offensive things just to see if it'll snap out of character or start defending itself. It doesn't. It answers plainly and moves on. Someone also asked one of the four if he knew about a modern cartoon character. He said he didn't, no guessing, no pretending to be current. What I like about all of this is that the system has no idea which questions are jokes and which are serious. It runs every one through the same steps, checks the same sources, follows the same rule about not saying things it can't back up. When something lands as funny, it's because the seriousness never slipped, not because I built a special case for jokes.

There's a heavier version of the same pattern too. Someone asked Jinnah, bluntly, why he ordered the Great Calcutta Killings. He did not. That's a real, common misattribution, and the answer correctly said so, cited, without getting defensive about it: a tragedy, not a planned action by him or the Muslim League. It's the kind of question where an answer that looks confident but isn't grounded in anything would do real damage, and this is exactly the case that made me build the citation checking in the first place.

The bug I only found by reading, not measuring

Here's the part I'd have missed entirely if I'd stuck to the dashboard. Out of those 805 answers, 7 of them, just under 1 percent, had a raw, broken citation tag sitting in the visible text instead of resolving into a clean footnote. Something like [[cite:Darwin's own words, 1876…]] showing up as literal text where a proper citation should have been.

The cause is a real streaming edge case. Citations get built by the model tagging a claim as it writes the answer live, word by word, and a separate piece of code catches that tag and swaps it for a clickable footnote before it ever reaches you. Most of the time that handoff is invisible. Seven times out of 805, the tag arrived split across two different chunks of the stream in a shape my catcher didn't expect, and the raw tag leaked through instead of getting cleaned up.

Under 1 percent is a genuinely small number, and I could have called it good enough and moved on. But a number on a dashboard doesn't tell you it looks like a broken robot mid-sentence to whoever actually saw it happen, and I'd rather know that than not. It's fixed now. I only knew to fix it because I went looking with my own eyes instead of trusting a pass rate that already looked fine.

Why this mattered more than the map does

I'll be honest, the countries and the funny questions were the fun part to write about, but the real value of that hour was in the boring part. Every real conversation is a free test of whether the citation rules hold up against a question I would never have thought to write myself. I can sit down and write test questions all day, but I can't predict the weird ones, and it turns out those tell you more about whether the guardrails actually work than the ones I'd write on purpose.

Not bad for something that started as a phrase stuck in my head at three in the morning and one very tired, very confused version of myself the next day. It's still not finished. There's more corpus to bring in, and I'm sure a careful enough reader will find question number eight that breaks something I haven't noticed yet. If you find it, there's a report button, and I read every one of those too, apparently including at three in the morning, when the questions get strangest.

Share this
X LinkedIn WhatsApp
Try Verbatim ↗