Polishing writing with AI · Part 1

Why AI-polished sentences give themselves away

·12 min read·한국어
ContentsThe more I polished, the more AI it soundedWhat the existing tools catch and missChanging where to stop and who checksA goal that is neither zero patterns nor a detector scoreWhy the side that did the edits shouldn't check themReal before and afterOpen problemsReferences

I write the posts on this blog together with an agent in the terminal. I built a polishing skill with the agent and went through the posts I had written so far, one at a time. It started with a feeling that my Korean posts read oddly like translations. When I counted across the 12 Korean posts, the ending ~것입니다 (-geosimnida, roughly 'it is that…') came up 79 times and ~셈입니다 (-semimnida, roughly 'it amounts to…') 16 times. Both are stiff, translated-sounding ways to close a Korean sentence. After the polish I counted again: 것입니다 was down to 13 and 셈입니다 to 0.

By the numbers it was a success. But when I read the posts again, a few of the edited spots read more like AI writing than before. This post is the record of finding out why and fixing the skill again.

The more I polished, the more AI it sounded

The Spotify post has a paragraph that carries one line from a 2014 video on Spotify's engineering culture. I wrote the English original in italics and glossed it in Korean right after.

Before the polish it looked like this.

다만 영상은 이 표 바로 다음에 대가도 함께 적었습니다. one consequence of autonomy is that we have very little standardization. 자율의 결과 중 하나로 표준화가 아주 적다는 것입니다.

In English: 'But right after this grid, the video also names the price. one consequence of autonomy is that we have very little standardization. It is that, as one consequence of autonomy, there is very little standardization.' The last sentence ends in ~다는 것입니다 (-daneun geosimnida), which works like a reported 'that is': it passes on what someone else said.

After the polish (commit 2d5d66f), only that last sentence changed.

자율의 결과 중 하나로 표준화가 아주 적다는 뜻입니다.

Now it ends in ~다는 뜻입니다 (-daneun tteusimnida), 'which means': 'Which means that, as one consequence of autonomy, there is very little standardization.'

The same sentence from the Spotify post, before and after the polish, stacked top to bottom. At the top is the video's line in English, one consequence of autonomy is that we have very little standardization. Below it, the Original row ends '...jeokda-neun geos-imnida.', glossed as '...is that standardization is very low', and the Polished row ends '...jeokda-neun tteus-imnida.', glossed as '...which means standardization is very low'. The changed syllables are highlighted with the note 'one syllable changed'. The Original row is tagged 'speaker: the video' and 'passes on what the video said'; the Polished row is tagged 'speaker: me' and 'reads as my own conclusion'.

~다는 것입니다 is translationese, but it still reads as an ending that relays someone else's words. Changed to ~다는 뜻입니다, it became a sentence in which I explain what the video said. The video is the one that said there was little standardization, and I was only passing that on, yet going by the sentence alone, the speaker had become me. While writing this post I changed that sentence to 표준화가 아주 적다고 했습니다 ('it said there is very little standardization').

It did not happen just once. When I counted the 12 Korean posts again after the polish, ~다는 뜻입니다 came up 13 times, and 5 of those were in the Spotify post. As I see it, another formula moved into the places where 것입니다 had been removed.

So my answer to the question in the title, for now, has two parts.

First, a polish that deletes patterns fills the gaps with new patterns. Delete 것입니다 and the sentence needs some other way to end, and the model picks the safest ending. In my posts that was a summing-up formula like ~다는 뜻입니다. The habit of evening out sentence lengths and adding a short wrap-up sentence at the end of a paragraph works in the same direction.

Second, when the side that did the edits checks its own work, those new patterns are hard to see. The first skill also had a step that matched each edited sentence against the original one by one, but the agent that did the edits ran that check itself. The Spotify sentence passed it, and it was not caught in the final review that day either.

With other tools, too, polishing made the text sound more like AI. Of the English polishing skills I looked at, the one with the most stars is humanizer, and Ryan Lenk posted the results of a blind comparison as issue #229. Language-model judges compared two texts without being told who wrote them, and the humanizer rewrite was rated better than the raw model output 16 times out of 16. But when the tells the judges cited as AI were counted by type, the results split.

Three arrows, each running from a start point to an end point, showing how many times LLM judges cited each tell before and after the humanizer rewrite. AI vocab drops from 70 to 0, marked 'removed'; uniform rhythm rises from 60 to 101 (+41) and aphoristic closers from 38 to 61 (+23), both marked 'went up'. Below is the source, blader/humanizer issue #229 by Ryan Lenk, with a note that in the same run, typography tells went from 60 to 0.

The tells on the word list disappeared, and the rhythm tells, which are not on any list, went up. The person who ran the experiment read it this way. My read is that a subtractive rewrite makes prose more uniformly well-shaped, and uniform good shape is itself a tell. The experimenter also noted the limits: the judges were all language models from the same family, and only 77 of the 144 planned runs were done.

What the existing tools catch and miss

Before fixing the skill again, I went through the polishing skills and style checkers out there with the agent.

I looked at several Korean skills with fewer than 100 stars. humanizer-ko groups summing-up formulas, ~다는 뜻이다 ('which means') among them, into their own set, and that is where my skill's rule on summing-up formulas comes from.

A matrix with tools down the side and eight columns across: Korean calques, AI vocab and clichés, rhythm and structure, meaning guard, machine check, regression tests, overfix alarm, and fresh checker. The tools are anthropics/skills, blader/humanizer, stop-slop, im-not-ai, Vale + vale-ai-tells, avoid-ai-writing, write-good, alex and proselint, unslop, patina, and at the bottom this blog's polish-post. Each cell is marked 'covers it', 'partly', 'misses it, or says it lacks it', or left blank for no evidence in my notes. The humanizer rhythm cell reads 60→101.

Most of them count or delete items from a list of words and phrases. They are good at catching AI vocabulary, dashes and clichés. Tells that come from the text as a whole, like rhythm, were often there only as a rule sentence such as 'vary sentence length'. About the only ones that actually measure it were unslop, which reports the variance of sentence length, and im-not-ai, which warns when parallel and contrast structures all disappear at once.

Safeguards for meaning were rarer still. humanizer says to keep every claim in the original, yet it allows the model to add concrete details it believes to be true. On my blog, unverified facts slipping in is what I guard against most, so I did not follow that part. In this area im-not-ai was the furthest ahead. It has a ledger that writes down the key nouns before editing and reverts if any of them go missing, a check for lost hedges and obligations, and thresholds that warn when the character change rate passes 30% and stop when it passes 50%. My skill's change-rate thresholds and a few of its warnings were inspired by it.

Changing where to stop and who checks

A goal that is neither zero patterns nor a detector score

First I had to change the goal of the skill I was fixing. The first skill had no standard for how far to cut, so it only ever moved toward deleting a signal whenever it saw one.

Another option was to aim at an AI detector score, but I decided not to use one. Liang et al. (2023) reported that seven detectors misclassified TOEFL essays by non-native writers as AI-written 61.22% of the time on average. Rewriting sentences to lower a score ends up throwing facts away.

The goal I chose instead is the range of my own writing. I measure how many times each pattern appears per 1,000 characters in the published posts on this blog, and stop once the edited post falls inside that range. Give the comparison script --baseline and it shows the range. Below is the output of a run I actually did while writing this post, with only the non-zero lines kept. The header reads 'author baseline (ko, 11 published posts, p10 · median · p90 per 1,000 characters)', and each line is a Korean pattern: 것입니다 'it is that', 에 대한·에 대해 'about', 것이었습니다 'it was that', 다는 뜻입니다·라는 뜻입니다 'which means', 다는 점입니다 'the point is that', 로 이어집니다 'leads to', and 어미 뒤 쉼표, a comma after a connective ending.

$ node scripts/polish-compare.ts --baseline ko
작성자 기준선 (ko, 공개 글 11편, 1000자당 p10 · 중앙값 · p90)
  것입니다: 0 · 0 · 0.25
  에 대한: 0 · 0 · 0.15
  에 대해: 0 · 0 · 0.11
  것이었습니다: 0 · 0 · 0.07
  다는 뜻입니다: 0 · 0.1 · 0.16
  라는 뜻입니다: 0 · 0 · 0.11
  다는 점입니다: 0 · 0 · 0.15
  로 이어집니다: 0 · 0 · 0.1
  어미 뒤 쉼표: 4.82 · 5.55 · 6.97

The other 21 patterns (셈입니다 and so on) were 0 on all three values. The 1,000 characters exclude spaces, and the post being compared is left out of the baseline. Among the phrase patterns, 다는 뜻입니다 (0.1) is the only one whose median is not 0. By the baseline, too, the summing-up formula from the start of this post is the habit most commonly left in my writing now. Commas after connective endings (-고, -며, -지만, and the like, roughly 'and,', 'while,', 'but,') are a different kind of thing, and my range is around 5 per 1,000 characters.

Two rows drawing the author baseline as bands, each measured across 11 published Korean posts per 1,000 non-space characters. The top row is the 'which means' ending -daneun tteusimnida, with a band at p10 0, median 0.1 and p90 0.16; to its right is 'under-polished, above the range', and a note says that since p10 is 0, there is no over-correction alarm. The bottom row is the comma after a connective ending, with a band at p10 4.82, median 5.55 and p90 6.97; to the right is 'under-polished', and to the left, below half of p10 (2.41), is 'over-correction'.

Above the range means it was under-polished, and inside the range means I do not cut any further. If a pattern falls below half of the range's lower end (p10), the script raises [경고] overcorrect ([warning] overcorrect). The warning tells me to look again at the deleted spots for any case the rules say to keep.

Why the side that did the edits shouldn't check them

Panickssery, Bowman, Feng (NeurIPS 2024) showed that language models acting as evaluators recognize their own writing and rate it higher. The better a model was at recognizing its own text, the more it preferred it. If that holds, asking the agent that did the edits to review the result means using a judge who goes easy on its own work.

This blog has a record of it too. In the final review on the day of the polish, I reverted sentences whose meaning had drifted (commit 5d6c813). In the candy-like blocks post, 무늬도 같은 비율로 줄인 셈입니다 ('it amounts to shrinking the pattern by the same ratio too') became 무늬도 줄였습니다 ('the pattern was shrunk too'), and 'by the same ratio' was lost. In the Call of Duty post, 다른 점은 ('the difference is') became 다만 ('however'), and a contrast that was not in the original appeared. Both sentences got past the self-review of the agent that did the edits.

So I handed the meaning check to a fresh agent. It works like this.

  1. Only the changed sentences are paired with the originals and numbered. Which one is the original is not marked; they are only A and B. For a deleted sentence, B is '(deleted)'.
  2. A new subagent gets only those pairs and three questions. If possible it runs on a different model from the one that did the polish. It is not told about the rule files, the post title, or that a polish happened.
  3. The questions are: is it the same claim (facts, numbers, subjects, causes), the same certainty (does a guess, possibility or obligation appear on only one side), and the same speaker (whose words or judgment is it). It is not asked whether the sentence reads well.
  4. Pairs reported as drifted are reverted to the original or fixed only where they drifted, and the fixed pairs are judged again by yet another fresh agent.

A flow diagram of the meaning check. On the left, the agent that polished knows the original, the rule files, the title and context, and that it did the edits, so it does not check itself: it rates its own text higher (self-preference bias). In the middle are only the changed sentences, as numbered A/B pairs; pair 2 is '...that it is low.' and '...which means it is low.' Which one is the original is not said. On the right, the context-free checker is a fresh agent, ideally another model, and gets only the A/B pairs and three questions (same claim, same certainty, same speaker), without the rule files, the post title, or the fact that a polish happened. It does not judge which reads better. It returns only the drifted numbers, for example '#2 | speaker | the video's words read as mine', and only #2 is reverted to the original.

If the Spotify sentence from the start went through this, it should fail the third question. A relays the video's words, and B is me explaining them. I have not yet measured whether the context-free side actually catches that difference.

Real before and after

While fixing the skill, I built a regression corpus by pulling before-and-after pairs out of the polishing commits in the git history. It covers the Korean and English versions of 11 posts, 22 pairs in all. Nine of those pairs also have a separate version corrected in 5d6c813 (fixed). The tests check that the accepted polishes have no [오류] ([error]), and that the warnings catch the two drifts fixed in 5d6c813 that a machine can catch.

The first example is the Call of Duty post.

  • Before the polish: 다른 점은 이제 픽셀마다 이웃을 모른 채 제 몫을 따로 정한다는 것입니다. ('The difference is that now each pixel decides its own share separately, without knowing its neighbors.')
  • After the polish: 다만 이제는 픽셀마다 이웃을 모른 채 제 몫을 따로 정합니다. ('However, now each pixel decides its own share separately, without knowing its neighbors.')
  • Corrected version: 달라진 점은 이제 픽셀마다 이웃을 모른 채 제 몫을 따로 정한다는 것입니다. ('What changed is that now each pixel decides its own share separately, without knowing its neighbors.')

Run the before and after through the current comparison script and you get the output below. [경고] 2건 means two warnings; 사라짐 is removed and 생김 is added, and 대조 is contrast. 패턴 is the pattern table, 변경률 the change rate, and 문장 29개 중 6개 means 6 of 29 sentences.

$ node scripts/polish-compare.ts tests/fixtures/polish/where-call-of-duty-draws-harder/ko/before.mdoc tests/fixtures/polish/where-call-of-duty-draws-harder/ko/after.mdoc
[경고] 2건
  number: 사라짐: "1/16" / 생김: "2", "16", "1", "1"
  connective: 다른 점은 이제 픽셀마다 이웃을 모른 채 제 몫을 따로 정한다는 것입니다. → 다만 이제는 픽셀마다 이웃을 모른 채 제 몫을 따로 정합니다. (생김: "대조")

패턴 (전 → 후, 1000자당 · 작성자 p10–p90)
  것입니다: 5 → 0 (4.67 → 0 · 0–0.25)
  에 대한: 2 → 2 (1.87 → 1.88 · 0–0.15)
  어미 뒤 쉼표: 7 → 7 (6.54 → 6.57 · 4.82–6.97)
어미 뒤 쉼표/문장: 0.24 → 0.24

변경률: 20.7% (문장 29개 중 6개)

connective is the warning that a cause or contrast not in the original has appeared. Compare against the corrected version and this warning goes away, leaving only the number warning from the edit that spelled out '½ to 1/16' as Korean fractions read aloud.

The second example is the Spotify post before and after the polish. The current script shows the problem from the start of this post in numbers. Its first line, 바뀌면 안 되는 것: 문제 없음, reads 'things that must not change: no problems'.

$ node scripts/polish-compare.ts tests/fixtures/polish/spotify-startup-spirit/ko/before.mdoc tests/fixtures/polish/spotify-startup-spirit/ko/after.mdoc
바뀌면 안 되는 것: 문제 없음

패턴 (전 → 후, 1000자당 · 작성자 p10–p90)
  것입니다: 9 → 0 (2.15 → 0 · 0–0.25)
  셈입니다: 3 → 0 (0.72 → 0 · 0–0)
  다는 뜻입니다: 0 → 2 (0 → 0.48 · 0–0.16)
  라는 뜻입니다: 1 → 1 (0.24 → 0.24 · 0–0.11)
  어미 뒤 쉼표: 21 → 21 (5.03 → 5.04 · 4.82–6.97)
어미 뒤 쉼표/문장: 0.26 → 0.26

변경률: 23.2% (문장 82개 중 19개)

것입니다 came down from above the range to 0, and 다는 뜻입니다 rose from 0 to 2, above the range (0.48). Yet the first line says 'no problems'. The machine check did not catch the change of speaker.

The same goes for 'by the same ratio' in the candy-like blocks post. Put it through the same script and you get 바뀌면 안 되는 것: 문제 없음. The missing words are an ordinary adverbial phrase, and warning every time an adverbial phrase goes missing would bury the report in warnings. In the test file too, I noted that drifts like this one, the choice of particle, and subject-predicate agreement cannot be caught with regular expressions, so they are left to the context-free checking agent.

Open problems

These are things I have not settled, or settled without being sure.

Where the baseline starts. The 11 published posts the baseline was measured on have all been through one polish. So this range is not the range of my writing before polishing, but of writing I accepted after polishing. I think the median of 다는 뜻입니다 not being 0 may also be a trace the polish left behind.

Update (November 15, 2026): commas dropped the baseline. The worry above showed up first in commas. Rereading my Korean posts, I found far too many commas, and the baseline itself leaned toward the comma-heavy side. So for commas alone I set a target instead of the baseline: at most 30% of sentences may contain a comma, and a sentence may hold more than one only when it lists three or more items. Applied to all 15 published posts, the share of sentences with a comma fell from 30–46% to 8–19% per post. Most fixes simply dropped the comma; about 50 sentences were split in two.

How much the checking agent helps. The procedure for polishing the corpus with the old and new versions of the skill and comparing them blind is written into the skill, but I have not run it yet. That is also when I will find out how many of the drifts in 5d6c813 the checking agent actually catches.

Back to the opening question, then: I found two answers. Whatever gets deleted is filled by new patterns, and the side doing the edits struggles to see those new patterns. The first can now be counted against the baseline of my own writing. The second I've handed to a checking agent with no context, but how much that check actually catches I'll only know once I run the blind comparison above.

References

Polishing and AI-style skills

Prose linters

AI detectors and evaluation research

Spelling