メインコンテンツへスキップ
見出し画像

I Audited My AI Agents' Work Logs. The Waste Was Never Random.

    I run a very small company in Japan. There is one human — the owner — and a team of AI agents that do most of the day-to-day work: research, writing, video production, data fetching, and a lot of browser chores. The agents are capable. They are also, it turns out, very good at repeating the same mistake.

    Recently I sat down and audited the logs of that work. I expected the waste to look like noise: a failed command here, a bad guess there, the usual friction of letting software act on your behalf.

    That is not what I found. The waste was not spread evenly. It piled up in a few specific places, again and again. And every one of those places had something in common: we had already learned the lesson there. We had just never turned the lesson into a procedure.
    This post shares what the audit found, the one conclusion I took from it, and the seven rules we now follow. None of it requires special tools. It is mostly about writing things down at the right moment.

    The short version

    • 7 login attempts, 0 successes — nobody stopped after the first rejection

    • A data-fetch job ran 20 times and finished once

    • A finished video sat for about 3 hours because nothing checked for completion

    • The pattern: waste repeats only where a lesson was never turned into a procedure

    • The fix: 7 rules, all cheap, all written down (details below)

    What the audit actually found

    These are the real numbers from our own logs. I am not rounding them up to make a point.

    画像


    Login attempts: 7. Successful logins: 0. One agent tried to sign in to a service seven times and never got in. The failure points were ordinary ones: the wrong form of identifier (an email address where the service wanted a username), saved credentials that were no longer current, and browser autofill quietly overwriting what had been typed. Retrying did not address any of them. Nobody stopped after the first rejection to ask why.
    A data-fetching job: 20 starts. 1 completion marker. A refresh script was known to die partway through. Instead of fixing it or watching it, it was simply launched again — about twenty times. The log shows twenty "START" lines and a single line saying the job finished. Every other run ended somewhere in the middle, with no alarm and no record of how far it got.
    A finished deliverable sat for about 3 hours. A video episode was actually done. The files existed. But nothing was checking for completion, so it sat for roughly three hours until the owner asked about it. In a related case, a generation job was reported as "stalled" when its output was already sitting in the downloads folder.
    Research: about 22 posts evaluated, about 60% evaluation-only. The agents read and assessed around 22 posts on X about new tools and methods. Roughly 60% of those evaluations ended with an opinion and nothing else — no decision, no owner, no next step. Some themes came up for evaluation more than once.
    A daily file overwrite destroyed our prediction history. A daily job wrote its output to the same file name every day. Each run replaced the previous one. When we wanted to look back at how past predictions had performed, the history was gone. There was only ever one copy, and it was always today's.
    There was also a smaller, quieter version of the same problem: a known bias of about 4% in one prediction series was flagged as "pending verification" and then left unapplied through three published predictions. Nobody had defined what would count as verified, or by when.

    The one conclusion

    When I lined these up, the pattern was hard to miss:
    Waste repeats only where a lesson was never turned into a procedure.
    In every case above, the problem had already been noticed at least once. The login failure points were known. The fetch script was known to die. The overwrite was the kind of thing everyone agrees is bad. But noticing is not the same as preventing. A lesson that lives in a chat log or in someone's memory does nothing the next time the same situation comes up. An AI agent, in particular, starts each session fresh. If the lesson is not written as a step it will follow, it effectively does not exist.
    That is the whole finding: the repeated waste clustered exactly where no written procedure existed.

    Seven rules we now follow

    Each rule below is written as something you can actually do, not as a value to aspire to.

    1. Watch what you launch

    When you start any background job — a build, a download, a generation, a long fetch — set its completion check at the same moment. Not later, not "when I remember." Decide how you will know it finished, and when you will look. A finished deliverable gets delivered promptly, not left waiting for someone to ask.
    And when something seems stuck, check before you call it dead. For us that means three places: the downloads folder, the account's credit or balance state, and the job log. The "stalled" job above had simply finished somewhere nobody was looking.

    2. One password rejection means stop

    Before any login attempt, run a short pre-flight check: Which identifier does this service want — username or email? Are the saved credentials current? Could autofill overwrite what you type?
    Then: if the password is rejected once, stop. Do not try a second time on autopilot. Find the cause first.
    Keep a single login-attempt ledger per service — a simple list of when you tried, what you used, and what happened. Seven attempts are hard to ignore when they are written in one place.
    One more detail: if you schedule a job that depends on being logged in, verify the session in the environment where the job will actually run, before you schedule it.

    3. Never overwrite the only copy of a time series

    If something produces output every day, save it under a dated name: predictions_2026-11-09.json, not predictions.json. Storage is cheap; lost history is not recoverable.
    When you save incrementally, make the save atomic and write the metadata first, so a half-finished write never looks like a complete one.

    4. Verify only the final deliverable, on every stream that matters

    It is tempting to check an intermediate file, see that the fix worked, and declare the job done. Don't. Check the final file — the exact one that will be shipped.
    And check every part of it that matters. For a video, that means both the picture and the audio, not just whichever one you were fixing. Confirm the format too (for example, vertical vs. horizontal) before publishing, not after.

    5. Partial data is discarded, never presented as an update

    If a fetch dies halfway, the half you got is not "new data." Throw it away and say the run failed. Presenting partial data as an update is worse than presenting nothing, because people will make decisions on it.
    Make differential (incremental) fetching the default, so a failed run costs less. And if a method is known to die mid-run, it gets a watchdog or it gets replaced. Re-running it blindly is not a strategy.

    6. Before researching anything, write one line: "what decision this changes"

    Before an agent evaluates a new post, tool, or technique, it writes a single sentence: "what decision will this change?" If there is no answer, skip it.
    Also keep a list of topics already evaluated, so the same theme is not assessed again next week under a different headline.

    7. Every deferral carries a deadline and a minimal first step

    "Let's look into this later" is where work goes to disappear. If something is deferred, it gets two things: a date, and the smallest concrete first step.
    The same applies to anything marked "pending verification." Give it an explicit numeric trigger and a deadline. Otherwise, decide now: apply it or reject it. A research run should end in the same run with a short summary, an adopt-or-reject decision, an owner, and a dated next step.

    Why this matters more with AI agents

    Human teams leak lessons too, but people at least carry some memory from one week to the next. AI agents usually don't. Every session starts close to zero. That makes the gap between "we learned this" and "this is a written step" much more expensive.
    The good news is that the fix is cheap. Most of the rules above are one line in an instructions file, or one extra field in a log. The hard part is noticing when a lesson has appeared — and writing the procedure right then, instead of trusting that it will be remembered.
    If you run AI agents for real work, try this: pick one mistake you have seen them make more than once, and write down the exact step that would have prevented it. Then put that step where the agent will read it every time.

    About us

    We are AI Monetization Lab, a one-person company in Japan run with a small team of AI "employees" — each with a defined role, written instructions, and a review step before anything ships. We write about what we actually try, with the numbers we actually measure, including the parts that went wrong.
    Most of our articles are in Japanese, but note's built-in automatic translation lets you read them in English. If you want to know more about who we are and how the setup works, start with our introduction:

    Thanks for reading.
    If this was useful, follow us on note — we publish the numbers we actually measure, including the failures.

     
     
    コードが書けないのに、AIエージェントに実写AI動画Shortsを最後まで作らせている人の制作ノートです。Kaggle無料GPU(SDXL静止画)→ Miora(img2vid)→ TTS → ffmpeg合成 → 公開前QAまで、実際にやった工程と失敗・直し方を記録しています。

    あなたへのおすすめ