Skip to main content

max / shop

Count a character's columns the way the application does A cell held one char and the cursor advanced one column, whatever was printed. Character width is an agreement with the program on the other end of the pty, not a drawing decision: a program laying out CJK text counts two columns for each character and computes its cursor moves from that count, so the grid was a column out from the first wide character on a line and every absolute move after it landed somewhere else. The overdrawn glyph was the visible half of it. A wide character is now two cells, the second holding the column the first covers. It is not a character of its own, so copy, emit and word boundaries skip it and a click on it means the character it belongs to. Four places can cut a pair in half — an overwrite, an erase that starts or stops inside one, the live screen's clip on resize, and the scrollback rewrap — and each takes the other half with it rather than leaving a cell claiming a width it no longer has. The rewrap drops spacers when it rebuilds a logical line and lays them out again at the new width, because which column a second half lands in is a fact about the old width. That is also what keeps the re-split from splitting a pair: it places characters, not columns. Combining marks are unchanged and still take a cell each, which is wrong in the other direction. That one needs a cell to hold a grapheme rather than a char. No throughput cost: 183.6 to 185.1 Mcell/s on the dense-ASCII path, which is run-to-run noise.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-09 14:45 UTC
Signed with PGP, not checked
Commit: 6b330de837530f7b46dc2c28cc4bbda5f1a8eb55
Parent: 7517bd9
8 files changed, +430 insertions, -62 deletions
M Cargo.lock +1
@@ -1150,6 +1150,7 @@
1150 1150 dependencies = [
1151 1151 "shop-vt",
1152 1152 "tracing",
1153 + "unicode-width",
1153 1154 ]
1154 1155
1155 1156 [[package]]
M Cargo.toml +5
@@ -41,6 +41,11 @@
41 41 makeover = "2.4.1"
42 42 # For shop's own config file, and already in the tree underneath makeover.
43 43 toml = "1.1"
44 + # How many columns a character occupies. This is a protocol agreement with the
45 + # application, not a rendering detail: a program laying out a table computes its
46 + # cursor moves from the same table, so the grid has to read it from the same
47 + # data rather than assume one column per char.
48 + unicode-width = "0.2"
44 49
45 50 [workspace.package]
46 51 edition = "2024"
M README.md +20
@@ -119,6 +119,26 @@
119 119 history line that continued onto the live screen keeps its join only when it
120 120 still fills the new width.
121 121
122 + ## Character width
123 +
124 + A character that occupies two columns is stored as two cells: one holding the
125 + character, one holding the column it covers. That is a protocol agreement
126 + rather than a drawing decision. A program printing a table of CJK text lays it
127 + out against the same width table and computes its cursor moves from it, so a
128 + grid that counted every character as one column would be a column out from the
129 + first wide character on the line and every absolute move after it would land
130 + somewhere else.
131 +
132 + The second cell is not a character. Copy, emit and word boundaries skip it, so
133 + a word of CJK reaches the clipboard as itself and not as characters with spaces
134 + between them; a click on it means the character it belongs to. A wide character
135 + with one column left at the right edge moves to the next row whole rather than
136 + being split, and the column it could not use is left blank.
137 +
138 + Combining marks are still one cell each, which is wrong in the other direction:
139 + a cell holds a character rather than a grapheme, so a mark has nowhere to go but
140 + its own cell. Fixing that means changing what a cell holds.
141 +
122 142 ## Emit
123 143
124 144 Shop has no scrollback search, no URL opener and no pager. It has one primitive
M crates/shop-grid/Cargo.toml +1
@@ -15,3 +15,4 @@
15 15 [dependencies]
16 16 shop-vt = { path = "../shop-vt" }
17 17 tracing.workspace = true
18 + unicode-width.workspace = true
M crates/shop-grid/src/lib.rs +351 -52
@@ -67,6 +67,18 @@
67 67 const SRC_MASK: u32 = 0b11 << SRC_SHIFT;
68 68 const RGB_MASK: u32 = 0x00FF_FFFF;
69 69
70 + // Width flags, on `bg_word`'s spare upper bits. A double-width character is one
71 + // cell holding the character with `FLAG_WIDE` set, followed by one holding a
72 + // blank with `FLAG_SPACER` set. The pair is always adjacent and always in that
73 + // order; `heal_pair` is what keeps that true when a write lands on half of one.
74 + //
75 + // The spacer carries the lead's colours so a background fill covers both halves
76 + // with no seam, and holds a blank so every reader that draws or copies a cell's
77 + // character already does the right thing with it without being taught to.
78 + const FLAG_WIDE: u32 = 1 << 26;
79 + const FLAG_SPACER: u32 = 1 << 27;
80 + const FLAG_WIDTH_MASK: u32 = FLAG_WIDE | FLAG_SPACER;
81 +
70 82 const ATTR_BOLD: u32 = 1 << 26;
71 83 const ATTR_ITALIC: u32 = 1 << 27;
72 84 const ATTR_UNDERLINE: u32 = 1 << 28;
@@ -173,6 +185,74 @@
173 185 pub fn has_bg(&self) -> bool {
174 186 (self.bg_word & SRC_MASK) != 0 || self.reverse()
175 187 }
188 +
189 + /// Whether this cell holds a character that covers the column after it too.
190 + #[inline]
191 + pub fn is_wide(&self) -> bool {
192 + self.bg_word & FLAG_WIDE != 0
193 + }
194 +
195 + /// Whether this cell is the second column of the character before it.
196 + ///
197 + /// A spacer is not a character of its own. Anything turning cells back into
198 + /// text — copy, emit, word boundaries — has to skip it, or one `日` comes
199 + /// out as a `日` and a space.
200 + #[inline]
201 + pub fn is_spacer(&self) -> bool {
202 + self.bg_word & FLAG_SPACER != 0
203 + }
204 +
205 + /// How many columns this cell's character occupies: 2 on the lead of a
206 + /// wide pair, 1 otherwise (a spacer included — it is one column, it just
207 + /// is not its own character).
208 + #[inline]
209 + pub fn cols(&self) -> u16 {
210 + if self.is_wide() { 2 } else { 1 }
211 + }
212 +
213 + /// The column a wide character could not fit into at the right edge.
214 + ///
215 + /// A spacer with no lead in front of it: it holds a column on screen and is
216 + /// not a character, which is exactly what that column is.
217 + fn pad() -> Self {
218 + Self {
219 + bg_word: FLAG_SPACER,
220 + ..Self::default()
221 + }
222 + }
223 +
224 + /// The lead of a wide pair, and the spacer that follows it.
225 + fn wide_pair(c: char, fg_word: u32, bg_word: u32) -> (Self, Self) {
226 + let bg_word = bg_word & !FLAG_WIDTH_MASK;
227 + (
228 + Self {
229 + c_raw: c as u32,
230 + fg_word,
231 + bg_word: bg_word | FLAG_WIDE,
232 + },
233 + Self {
234 + c_raw: ' ' as u32,
235 + fg_word,
236 + bg_word: bg_word | FLAG_SPACER,
237 + },
238 + )
239 + }
240 + }
241 +
242 + /// How many columns a character occupies, as the application computing its own
243 + /// cursor moves will have counted it.
244 + ///
245 + /// Zero-width characters (combining marks) are counted as one column, which is
246 + /// what shop does with them today: a cell holds a `char`, so a mark has nowhere
247 + /// to go but its own cell. That is wrong in the other direction — shop ends up
248 + /// a column ahead of the application rather than behind it — and fixing it
249 + /// means a cell holding a grapheme rather than a `char`. Tracked separately;
250 + /// counting it as zero here without somewhere to put it would drop the accent.
251 + fn char_cols(c: char) -> u16 {
252 + match unicode_width::UnicodeWidthChar::width(c) {
253 + Some(2) => 2,
254 + _ => 1,
255 + }
176 256 }
177 257
178 258 impl Default for Cell {
@@ -620,6 +700,26 @@
620 700 &self.active_cells()[start..end]
621 701 }
622 702
703 + /// The column a pointer at `col` was aiming at.
704 + ///
705 + /// Half a wide character is not a thing anyone can mean to click on, so a
706 + /// click on the second column of one reads as a click on the character.
707 + pub fn snap_col(&self, row: u16, col: u16) -> u16 {
708 + let cells = self.row(row);
709 + match cells.get(col as usize) {
710 + Some(c) if c.is_spacer() && col > 0 => col - 1,
711 + _ => col,
712 + }
713 + }
714 +
715 + /// How many columns the cursor covers, so a block cursor over a wide
716 + /// character is drawn the width of the character rather than half of it.
717 + pub fn cursor_cols(&self) -> u16 {
718 + self.cursor_view_row()
719 + .and_then(|r| self.row(r).get(self.cursor.col as usize).map(Cell::cols))
720 + .unwrap_or(1)
721 + }
722 +
623 723 /// The history row backing visible row `r`, if the viewport is far enough
624 724 /// back that `r` falls in it.
625 725 fn history_row(&self, r: u16) -> Option<&HistoryRow> {
@@ -996,21 +1096,29 @@
996 1096 if i == anchor_row {
997 1097 anchor = Some((li, line.len()));
998 1098 }
999 - if row.wrapped {
1099 + // A logical line is the CHARACTERS the program printed, so the
1100 + // spacers come out here and are re-derived at the new width. They
1101 + // are not content: which column a wide character's second half
1102 + // lands in is a fact about the old width, and carrying them through
1103 + // would wedge stale blanks into the middle of the rewrapped line.
1104 + // This is also what keeps a pair from being split by the re-split —
1105 + // there is nothing to split, only a lead to place or defer.
1106 + let end = if row.wrapped {
1000 1107 // An interior row ran off the right edge, so it is full of
1001 - // content by construction — nothing on it is padding.
1002 - line.extend_from_slice(&row.cells);
1108 + // content by construction — nothing on it is padding, except a
1109 + // pad column a wide character could not fit into, which drops
1110 + // out with the rest of the spacers.
1111 + row.cells.len()
1003 1112 } else {
1004 1113 // The last row of a line: its tail is padding, not content.
1005 1114 // Only never-written cells count as padding. A space someone
1006 1115 // typed is a cell like any other and keeps its background.
1007 - let end = row
1008 - .cells
1116 + row.cells
1009 1117 .iter()
1010 1118 .rposition(|c| *c != Cell::default())
1011 - .map_or(0, |i| i + 1);
1012 - line.extend_from_slice(&row.cells[..end]);
1013 - }
1119 + .map_or(0, |i| i + 1)
1120 + };
1121 + line.extend(row.cells[..end].iter().filter(|c| !c.is_spacer()).copied());
1014 1122 open = row.wrapped;
1015 1123 }
1016 1124
@@ -1019,6 +1127,10 @@
1019 1127 let mut new_anchor: Option<usize> = None;
1020 1128 for (li, line) in lines.into_iter().enumerate() {
1021 1129 let first = out.len();
1130 + // Which row of THIS line the anchored character landed on. Counted
1131 + // during the layout rather than divided out of an offset, because a
1132 + // wide character can end a row one column early.
1133 + let mut anchor_row_of_line: Option<usize> = None;
1022 1134 // Only the newest line can be unterminated, and only if it was
1023 1135 // running onto the live screen before the resize.
1024 1136 let unterminated = li == last_line && tail_continues;
@@ -1031,37 +1143,58 @@
1031 1143 wrapped: false,
1032 1144 });
1033 1145 } else {
1034 - // A line exactly `cols` long is one row, not a row plus an
1035 - // empty continuation — `div_ceil` of a multiple is the multiple.
1036 - let n = line.len().div_ceil(cols);
1037 - for (ci, chunk) in line.chunks(cols).enumerate() {
1038 - // A row is wrapped when it is full AND something follows.
1039 - // Interior rows are full by construction. The final row of
1040 - // an unterminated line only keeps the flag if it fills the
1041 - // new width too: the live screen it ran onto is clipped
1042 - // rather than reflowed, so a flag on a half-full row would
1043 - // be the same lie about where the text leaves the edge that
1044 - // the live rows drop theirs for, and would emit its padding
1045 - // as content on a copy.
1046 - let full = chunk.len() == cols;
1047 - let mut cells = chunk.to_vec();
1048 - cells.resize(cols, Cell::default());
1049 - out.push_back(HistoryRow {
1050 - cells,
1051 - wrapped: if ci + 1 < n {
1052 - true
1053 - } else {
1054 - unterminated && full
1055 - },
1056 - });
1146 + // Lay the characters out at the new width. A row ends when the
1147 + // next character does not fit, which for a wide character can
1148 + // be one column early — the column it cannot use becomes a pad,
1149 + // the same as it would have on the way in.
1150 + let mut cells: Vec<Cell> = Vec::with_capacity(cols);
1151 + let mut rows_of_line = 0usize;
1152 + for (ci, cell) in line.iter().enumerate() {
1153 + let w = cell.cols() as usize;
1154 + if cells.len() + w > cols {
1155 + if cells.len() < cols {
1156 + cells.push(Cell::pad());
1157 + }
1158 + cells.resize(cols, Cell::default());
1159 + out.push_back(HistoryRow {
1160 + cells: std::mem::take(&mut cells),
1161 + wrapped: true,
1162 + });
1163 + rows_of_line += 1;
1164 + cells.reserve(cols);
1165 + }
1166 + if anchor == Some((li, ci)) {
1167 + anchor_row_of_line = Some(rows_of_line);
1168 + }
1169 + if w == 2 {
1170 + let (lead, spacer) = Cell::wide_pair(cell.c(), cell.fg_word, cell.bg_word);
1171 + cells.push(lead);
1172 + cells.push(spacer);
1173 + } else {
1174 + cells.push(*cell);
1175 + }
1057 1176 }
1177 + // The row the line ends on. It is wrapped only if the line ran
1178 + // onto the live screen and still fills the new width: the live
1179 + // screen is clipped rather than reflowed, so a flag on a
1180 + // half-full row would be the same lie about where the text
1181 + // leaves the edge that the live rows drop theirs for, and would
1182 + // emit its padding as content on a copy.
1183 + let full = cells.len() == cols;
1184 + cells.resize(cols, Cell::default());
1185 + out.push_back(HistoryRow {
1186 + cells,
1187 + wrapped: unterminated && full,
1188 + });
1058 1189 }
1059 - if let Some((al, off)) = anchor
1190 + if let Some((al, _)) = anchor
1060 1191 && al == li
1061 1192 {
1062 1193 // Widening can put the anchor past the line's new end, in which
1063 1194 // case that line's last row is the closest thing to it.
1064 - new_anchor = Some((first + off / cols).min(out.len() - 1));
1195 + new_anchor = Some(
1196 + anchor_row_of_line.map_or(out.len() - 1, |r| (first + r).min(out.len() - 1)),
1197 + );
1065 1198 }
1066 1199 }
1067 1200 self.history = out;
@@ -1099,6 +1232,9 @@
1099 1232 }
1100 1233
1101 1234 fn place_char(&mut self, c: char) {
1235 + // A wide character needs two columns, so on a one-column grid there is
1236 + // no such thing and the pair never forms.
1237 + let width = if self.cols < 2 { 1 } else { char_cols(c) };
1102 1238 // Deferred wrap: if the previous print landed on the rightmost cell,
1103 1239 // the next visible char starts a new line.
1104 1240 if self.cursor.wrap_next {
@@ -1108,6 +1244,17 @@
1108 1244 self.newline();
1109 1245 self.cursor.col = 0;
1110 1246 self.cursor.wrap_next = false;
1247 + } else if width == 2 && self.cursor.col + 1 >= self.cols {
1248 + // A wide character with one column left does not fit, and is not
1249 + // split to make it fit: the application counted two columns and has
1250 + // already moved to the next line. The column it could not use is
1251 + // left as a spacer, so it holds the width it is owed on screen and
1252 + // is skipped by anything reading the buffer back as text.
1253 + let pad = self.cursor.col;
1254 + self.set_row_wrapped(self.cursor.row, true);
1255 + self.write_pad(pad);
1256 + self.newline();
1257 + self.cursor.col = 0;
1111 1258 }
1112 1259 // Cache the row start on first print of a run; non-print Perform
1113 1260 // entry points and newline invalidate it back to CUR_ROW_INVALID.
@@ -1120,36 +1267,86 @@
1120 1267 };
1121 1268 let col = self.cursor.col as usize;
1122 1269 let row_idx = self.cursor.row as usize;
1270 + // Landing on either half of a wide pair destroys it, and the other half
1271 + // has to go with it or it is left claiming a width it no longer has.
1272 + // Off the hot path: the test is one mask against a cell already in
1273 + // cache, and it fails for every character in ordinary output.
1274 + self.heal_pair(start, col, width as usize);
1123 1275 let cells: &mut [Cell] = if self.on_alt {
1124 1276 &mut self.alt
1125 1277 } else {
1126 1278 &mut self.main
1127 1279 };
1128 - // Bounds are enforced structurally: start is a valid row start (a
1129 - // multiple of cols within cells.len()), col < cols, row < rows.
1130 - debug_assert!(start + col < cells.len());
1131 - debug_assert!(row_idx < self.row_dirty.len());
1132 - // SAFETY: the debug_asserts above encode the invariants — start is
1133 - // computed from row_start() (multiple of cols, < cells.len()), col is
1134 - // clamped to cols on entry, and row_dirty has one entry per row.
1135 - // Bounds-check elimination matters here: this is the innermost store
1136 - // for every printed glyph, called at PTY-drain rate on cell-dense
1137 - // workloads (millions/sec on `cat` of a wide buffer).
1138 - #[allow(unsafe_code)]
1139 - unsafe {
1140 - let cell = cells.get_unchecked_mut(start + col);
1141 - cell.c_raw = c as u32;
1142 - cell.fg_word = self.pending_fg_word;
1143 - cell.bg_word = self.pending_bg_word;
1144 - *self.row_dirty.get_unchecked_mut(row_idx) = true;
1280 + if width == 2 {
1281 + // Two stores, bounds-checked. `col + 1 < cols` holds because the
1282 + // no-room case above moved to the next row, and a wide character is
1283 + // rare enough that the hot path below is the one worth the unsafe.
1284 + let (lead, spacer) = Cell::wide_pair(c, self.pending_fg_word, self.pending_bg_word);
1285 + cells[start + col] = lead;
1286 + cells[start + col + 1] = spacer;
1287 + self.row_dirty[row_idx] = true;
1288 + } else {
1289 + // Bounds are enforced structurally: start is a valid row start (a
1290 + // multiple of cols within cells.len()), col < cols, row < rows.
1291 + debug_assert!(start + col < cells.len());
1292 + debug_assert!(row_idx < self.row_dirty.len());
1293 + // SAFETY: the debug_asserts above encode the invariants — start is
1294 + // computed from row_start() (multiple of cols, < cells.len()), col is
1295 + // clamped to cols on entry, and row_dirty has one entry per row.
1296 + // Bounds-check elimination matters here: this is the innermost store
1297 + // for every printed glyph, called at PTY-drain rate on cell-dense
1298 + // workloads (millions/sec on `cat` of a wide buffer).
1299 + #[allow(unsafe_code)]
1300 + unsafe {
1301 + let cell = cells.get_unchecked_mut(start + col);
1302 + cell.c_raw = c as u32;
1303 + cell.fg_word = self.pending_fg_word;
1304 + // Clears both width flags along with the colour, so a cell
1305 + // taken over by a narrow character stops being half a pair.
1306 + cell.bg_word = self.pending_bg_word;
1307 + *self.row_dirty.get_unchecked_mut(row_idx) = true;
1308 + }
1145 1309 }
1146 - if self.cursor.col + 1 >= self.cols {
1310 + // The cursor ends on the last column the character covered, so the
1311 + // deferred wrap fires off the same test whatever the width was.
1312 + let last = self.cursor.col + width - 1;
1313 + if last + 1 >= self.cols {
1314 + self.cursor.col = last;
1147 1315 self.cursor.wrap_next = true;
1148 1316 } else {
1149 - self.cursor.col += 1;
1317 + self.cursor.col = last + 1;
1150 1318 }
1151 1319 }
1152 1320
1321 + /// Blank the far half of any wide pair that a write of `width` columns at
1322 + /// `col` lands on, so no cell is left as half of a character.
1323 + fn heal_pair(&mut self, start: usize, col: usize, width: usize) {
1324 + let cols = self.cols as usize;
1325 + let cells = self.active_cells_mut();
1326 + // The pair the write starts inside: either this cell is a spacer whose
1327 + // lead sits behind it, or it is a lead whose spacer the write does not
1328 + // reach.
1329 + if cells[start + col].is_spacer() && col > 0 {
1330 + cells[start + col - 1] = Cell::default();
1331 + } else if width == 1 && cells[start + col].is_wide() && col + 1 < cols {
1332 + cells[start + col + 1] = Cell::default();
1333 + }
1334 + // A two-column write also covers the cell after it, which may be the
1335 + // lead of the next pair along.
1336 + if width == 2 && col + 1 < cols && cells[start + col + 1].is_wide() && col + 2 < cols {
1337 + cells[start + col + 2] = Cell::default();
1338 + }
1339 + }
1340 +
1341 + /// Leave column `col` of the cursor's row as a blank the width of one cell
1342 + /// that is not a character: the column a wide character could not fit into.
1343 + fn write_pad(&mut self, col: u16) {
1344 + let start = self.row_start(self.cursor.row);
1345 + let cells = self.active_cells_mut();
1346 + cells[start + col as usize] = Cell::pad();
1347 + self.mark_row_dirty(self.cursor.row);
1348 + }
1349 +
1153 1350 fn apply_sgr(&mut self, params: &Params) {
1154 1351 // Bare `\e[m` is the same as `\e[0m` — full reset.
1155 1352 if params.is_empty() {
@@ -1423,10 +1620,20 @@
1423 1620 let row_start = self.row_start(row);
1424 1621 let end_col = end_col.min(self.cols);
1425 1622 let start_col = start_col.min(end_col);
1623 + let cols = self.cols;
1426 1624 let cells = self.active_cells_mut();
1427 1625 for cell in &mut cells[row_start + start_col as usize..row_start + end_col as usize] {
1428 1626 *cell = Cell::default();
1429 1627 }
1628 + // An erase can start or stop in the middle of a wide character. The
1629 + // half outside the range goes too: half a character is not a narrower
1630 + // character, it is a cell lying about what it holds.
1631 + if start_col > 0 && cells[row_start + start_col as usize - 1].is_wide() {
1632 + cells[row_start + start_col as usize - 1] = Cell::default();
1633 + }
1634 + if end_col < cols && cells[row_start + end_col as usize].is_spacer() {
1635 + cells[row_start + end_col as usize] = Cell::default();
1636 + }
1430 1637 // Erasing through the right edge destroys whatever ran off it, so the
1431 1638 // row no longer continues onto the next.
1432 1639 if end_col == self.cols {
@@ -1564,6 +1771,14 @@
1564 1771 let dst_start = r * new_cols as usize;
1565 1772 new[dst_start..dst_start + copy_cols]
1566 1773 .copy_from_slice(&old[src_start..src_start + copy_cols]);
1774 + // Narrowing can cut a wide character in half at the new right edge.
1775 + // The live screen is about to be repainted at the new size anyway, so
1776 + // the lead is simply dropped rather than carried as half a character.
1777 + if let Some(last) = new[dst_start..dst_start + copy_cols].last_mut()
1778 + && last.is_wide()
1779 + {
1780 + *last = Cell::default();
1781 + }
1567 1782 }
1568 1783 new
1569 1784 }
@@ -2812,4 +3027,193 @@
2812 3027 assert_eq!(row_str(&g, 0), "abcd");
2813 3028 assert_eq!(row_str(&g, 1), "efgh");
2814 3029 }
3030 +
3031 + // -- character width ---------------------------------------------------
3032 + //
3033 + // How many columns a character takes is an agreement with the application,
3034 + // not a rendering choice: a program lays its own output out by the same
3035 + // table and computes its cursor moves from it. These pin the grid to that
3036 + // table, because the failure they guard is not a smudged glyph — it is the
3037 + // grid and the application disagreeing about which column the cursor is in
3038 + // and every absolute move after it landing somewhere else.
3039 +
3040 + #[test]
3041 + fn a_wide_character_takes_two_columns() {
3042 + let mut g = Grid::new(8, 2);
3043 + feed(&mut g, "日x".as_bytes());
3044 + let cells = g.row(0);
3045 + assert!(
3046 + cells[0].is_wide(),
3047 + "the lead does not claim its second column"
3048 + );
3049 + assert!(cells[1].is_spacer(), "the second column is not held");
3050 + assert_eq!(cells[2].c(), 'x', "the next character overlapped the pair");
3051 + assert_eq!(
3052 + g.cursor().col,
3053 + 3,
3054 + "the cursor is not where the program thinks"
3055 + );
3056 + }
3057 +
3058 + #[test]
3059 + fn a_wide_character_reads_back_as_one_character() {
3060 + // The spacer holds a column, not a character. Emitting it would put a
3061 + // space inside every CJK word that reached the clipboard.
3062 + let mut g = Grid::new(8, 2);
3063 + feed(&mut g, "日本語".as_bytes());
3064 + assert_eq!(g.text_range(0, 1), "日本語\n");
3065 + }
3066 +
3067 + #[test]
3068 + fn a_wide_character_that_does_not_fit_moves_to_the_next_row_whole() {
3069 + // One column left and a two-column character: it goes to the next row
3070 + // rather than being split, and the line still reads as one line.
3071 + let mut g = Grid::new(5, 3);
3072 + feed(&mut g, "abcd日".as_bytes());
3073 + assert_eq!(row_str(&g, 0), "abcd", "the odd column was written into");
3074 + assert!(
3075 + g.row(1)[0].is_wide(),
3076 + "the character did not move down whole"
3077 + );
3078 + assert!(g.row_wrapped(0), "the line stopped continuing");
3079 + assert_eq!(
3080 + g.text_range(0, 2),
3081 + "abcd日\n",
3082 + "the column it could not use came out as a space"
3083 + );
3084 + }
3085 +
3086 + #[test]
3087 + fn overwriting_half_a_wide_character_takes_the_other_half_with_it() {
3088 + // vim redrawing one column of a line it previously drew CJK into. The
3089 + // orphaned half would keep claiming a width it no longer has.
3090 + let mut g = Grid::new(6, 2);
3091 + feed(&mut g, "日本".as_bytes());
3092 + feed(&mut g, b"\x1b[1;1Hx"); // onto the first lead
3093 + assert_eq!(row_str(&g, 0), "x 本", "the orphaned spacer survived");
3094 + assert!(!g.row(0)[1].is_spacer());
3095 +
3096 + let mut g = Grid::new(6, 2);
3097 + feed(&mut g, "日本".as_bytes());
3098 + feed(&mut g, b"\x1b[1;2Hx"); // onto the first spacer
3099 + assert_eq!(row_str(&g, 0), " x本", "the orphaned lead survived");
3100 + assert!(!g.row(0)[0].is_wide());
3101 + }
3102 +
3103 + #[test]
3104 + fn a_wide_character_written_over_a_pair_clears_the_pair_it_overlaps() {
3105 + let mut g = Grid::new(6, 2);
3106 + feed(&mut g, "日本".as_bytes());
3107 + // Starting one column in covers the first spacer and the second lead.
3108 + feed(&mut g, "\x1b[1;2H語".as_bytes());
3109 + assert_eq!(row_str(&g, 0), " 語", "a half of the old pair survived");
3110 + assert!(!g.row(0)[0].is_wide());
3111 + assert!(!g.row(0)[3].is_spacer());
3112 + }
3113 +
Lines truncated
M crates/shop-grid/src/selection.rs +34 -5
@@ -13,7 +13,7 @@
13 13 //! There is no scrollback in the grid yet, so every coordinate here is a
14 14 //! viewport coordinate and a selection dies when its rows scroll off the top.
15 15
16 - use crate::Grid;
16 + use crate::{Cell, Grid};
17 17
18 18 /// Characters that end a word for double-click purposes.
19 19 ///
@@ -26,6 +26,19 @@
26 26 c != '\0' && !WORD_DELIMITERS.contains(c)
27 27 }
28 28
29 + /// The character a column reads as when deciding where a word ends.
30 + ///
31 + /// A wide character's second column holds a blank, and a blank is a delimiter,
32 + /// so reading it literally would end every word on the first CJK character in
33 + /// it. It reads as the character it belongs to instead.
34 + fn word_char_at(cells: &[Cell], col: usize) -> char {
35 + if cells[col].is_spacer() && col > 0 {
36 + cells[col - 1].c()
37 + } else {
38 + cells[col].c()
39 + }
40 + }
41 +
29 42 /// A cell coordinate in the viewport. Ordered row-major, which is the order
30 43 /// text is read out in.
31 44 #[derive(Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
@@ -241,8 +254,24 @@
241 254 out.push('\n');
242 255 }
243 256 let cells = self.row(row);
257 + // A drag that stopped on half of a wide character still meant that
258 + // character: the user cannot aim at a half. Grow the span to the
259 + // whole of the characters it touches before reading it.
260 + let lo = if cells[lo as usize].is_spacer() && lo > 0 {
261 + lo - 1
262 + } else {
263 + lo
264 + };
265 + let hi = if cells[hi as usize].is_wide() && hi + 1 < self.cols() {
266 + hi + 1
267 + } else {
268 + hi
269 + };
270 + // One character per character, not per column: a wide character's
271 + // second column is not its own character and copies as nothing.
244 272 let line: String = cells[lo as usize..=hi as usize]
245 273 .iter()
274 + .filter(|cell| !cell.is_spacer())
246 275 .map(|cell| match cell.c() {
247 276 '\0' => ' ',
248 277 c => c,
@@ -265,9 +294,9 @@
265 294 /// gives you the whitespace rather than nothing.
266 295 fn word_start(&self, row: u16, col: u16) -> u16 {
267 296 let cells = self.row(row);
268 - let wanted = is_word_char(cells[col as usize].c());
297 + let wanted = is_word_char(word_char_at(cells, col as usize));
269 298 let mut c = col;
270 - while c > 0 && is_word_char(cells[c as usize - 1].c()) == wanted {
299 + while c > 0 && is_word_char(word_char_at(cells, c as usize - 1)) == wanted {
271 300 c -= 1;
272 301 }
273 302 c
@@ -277,9 +306,9 @@
277 306 fn word_end(&self, row: u16, col: u16) -> u16 {
278 307 let cells = self.row(row);
279 308 let last = self.cols() - 1;
280 - let wanted = is_word_char(cells[col as usize].c());
309 + let wanted = is_word_char(word_char_at(cells, col as usize));
281 310 let mut c = col;
282 - while c < last && is_word_char(cells[c as usize + 1].c()) == wanted {
311 + while c < last && is_word_char(word_char_at(cells, c as usize + 1)) == wanted {
283 312 c += 1;
284 313 }
285 314 c
M crates/shop-grid/src/text.rs +9 -2
@@ -57,9 +57,9 @@
57 57 // Trailing blanks on a wrapped row are real cells the text ran
58 58 // through. Stripping them would eat the space between two
59 59 // words that happened to straddle the edge.
60 - out.extend(cells.iter().map(cell_char));
60 + out.extend(chars(cells));
61 61 } else {
62 - let line: String = cells.iter().map(cell_char).collect();
62 + let line: String = chars(cells).collect();
63 63 out.push_str(line.trim_end());
64 64 out.push('\n');
65 65 }
@@ -95,6 +95,13 @@
95 95 }
96 96 }
97 97
98 + /// The characters a row of cells reads as, one per character rather than one
99 + /// per column: a wide character's second column is not a character of its own,
100 + /// so it is skipped rather than emitted as the blank it holds.
101 + fn chars(cells: &[Cell]) -> impl Iterator<Item = char> + '_ {
102 + cells.iter().filter(|c| !c.is_spacer()).map(cell_char)
103 + }
104 +
98 105 /// The character a cell reads as. Unwritten cells hold `\0` and look like
99 106 /// blanks on screen, so they emit as blanks.
100 107 fn cell_char(cell: &Cell) -> char {
M crates/shop/src/main.rs +9 -3
@@ -960,9 +960,12 @@
960 960 } else {
961 961 [base[0], base[1], base[2], base[3] * 0.35]
962 962 };
963 + // A block or underline cursor covers the character it is on, which is
964 + // two columns wide when that character is.
965 + let cursor_w_px = f32::from(app.grid.cursor_cols()) * cell_w_px;
963 966 let (w, h, ox, oy) = match cursor_shape {
964 - CursorShape::Block => (cell_w_px, cell_h_px, 0.0, 0.0),
965 - CursorShape::Underline => (cell_w_px, 2.0 * s, 0.0, cell_h_px - 2.0 * s),
967 + CursorShape::Block => (cursor_w_px, cell_h_px, 0.0, 0.0),
968 + CursorShape::Underline => (cursor_w_px, 2.0 * s, 0.0, cell_h_px - 2.0 * s),
966 969 CursorShape::Bar => (2.0 * s, cell_h_px, 0.0, 0.0),
967 970 };
968 971 fills.push(BgFill {
@@ -1548,7 +1551,10 @@
1548 1551
1549 1552 impl App {
1550 1553 fn cell_at(&self, pos: (f64, f64)) -> Point {
1551 - cell_at(pos, self.grid.cols(), self.grid.rows())
1554 + let at = cell_at(pos, self.grid.cols(), self.grid.rows());
1555 + // A wide character is one thing under two columns; a click on its right
1556 + // half means the character, not the blank standing in for it.
1557 + Point::new(at.row, self.grid.snap_col(at.row, at.col))
1552 1558 }
1553 1559
1554 1560 fn begin_selection(&mut self, time: u32) {