Skip to main content

max / makenotwork

s3-storage: a live suite against a real object store The crate had no integration test of any kind. A mutation run said what that costs: 114 of 206 mutants survived, 104 of them inside S3Client because nothing in the repo ever calls upload, download, delete, copy_object, delete_prefix, presign_download or the three multipart drivers (infra 1498fe2a, ruled 06cf56c9 as option (c)). Eleven tests, all #[ignore]d so a laptop reports them SKIPPED rather than passing silently, run against MinIO on astra under a bucket the credentials are scoped to (infra 6f7c7b46). Each works under a prefix carrying the clock, the pid and a counter, because a mutation run puts a dozen copies of this suite against one bucket at once. The one the file exists for: a multipart copy of ~11 MiB in 5 MiB parts, downloaded and compared byte for byte. A copy driver whose part offsets are off duplicates or drops a range, the object still completes, and nothing but the bytes says so. Same shape for upload_multipart, with a deliberately short final part. Presigned URLs are fetched over raw TCP rather than through an HTTP dependency: the tier's endpoint is plain HTTP on localhost, and adding an HTTP stack would drag a TLS backend into a crate that pins its own.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-27 02:50 UTC
Signed with PGP, not checked
Commit: adb5ec3fcf54c9731f04d03e46f2a6ba606da356
Parent: c20786c
1 file changed, +500 insertions, -0 deletions
A shared/s3-storage/tests/live_s3.rs +500
@@ -1,0 +1,547 @@
1 + //! Every `S3Client` method, against a real object store.
2 + //!
3 + //! WHY THIS FILE EXISTS. The crate had no integration test of any kind, and a
4 + //! mutation run said what that costs: 114 of 206 mutants survived, 104 of them
5 + //! inside `S3Client` (infra `1498fe2a`). `upload -> Ok(())` passed the whole
6 + //! suite, because nothing in this repo ever called `upload`. The two worth
7 + //! naming: `attempt < 3` can become `true` in all three multipart drivers,
8 + //! which is an infinite retry on a permanent failure, and the part-offset
9 + //! arithmetic in the copy driver can be wrong in either direction, which is
10 + //! silent corruption of creator media -- the object completes and the bytes are
11 + //! the wrong ones. Nothing short of a real round trip observes that.
12 + //!
13 + //! HOW TO RUN IT. The tier is astra, where MinIO is a local service:
14 + //!
15 + //! set -a; . ~/.config/s3-storage-tests.env; set +a
16 + //! cargo nextest run --run-ignored all
17 + //!
18 + //! Every test is `#[ignore]`d so a developer machine reports them as SKIPPED
19 + //! rather than passing silently. A test that did not run must never read as one
20 + //! that did, which is the same rule the sweep applies to a cell it could not
21 + //! evaluate.
22 + //!
23 + //! ISOLATION. `S3_TEST_*` names a bucket the credentials can reach and nothing
24 + //! else: the key is scoped to it and MinIO refuses a write to the media buckets
25 + //! (infra `6f7c7b46`, measured). Belt and braces anyway, every test works under
26 + //! its own unique prefix and deletes it on the way out, so two runs cannot
27 + //! collide and a leaked object is traceable to a run.
28 + //!
29 + //! MINIO IS NOT S3, and where they differ these tests assert the property that
30 + //! matters rather than the exact bytes of a header. Multipart ETags are the
31 + //! usual example: their format is not contractual, so nothing here reads one.
32 +
33 + use std::io::{Read, Write};
34 + use std::net::TcpStream;
35 + use std::sync::atomic::{AtomicU32, Ordering};
36 + use std::time::{SystemTime, UNIX_EPOCH};
37 +
38 + use s3_storage::{S3Client, S3Config};
39 +
40 + const SKIP: &str = "live S3: set S3_TEST_* and run with --run-ignored (see the module docs)";
41 +
42 + /// The bucket and credentials, or a panic naming what is missing.
43 + ///
44 + /// A panic rather than a skip: these tests are `#[ignore]`d, so the only way to
45 + /// reach this code is to have asked for them, and silently passing an asked-for
46 + /// test because a variable was unset is the failure this whole file is about.
47 + fn config() -> S3Config {
48 + fn var(name: &str) -> String {
49 + std::env::var(name)
50 + .unwrap_or_else(|_| panic!("{name} is not set; source ~/.config/s3-storage-tests.env"))
51 + }
52 + S3Config {
53 + endpoint: var("S3_TEST_ENDPOINT"),
54 + bucket: var("S3_TEST_BUCKET"),
55 + access_key: var("S3_TEST_ACCESS_KEY"),
56 + secret_key: var("S3_TEST_SECRET_KEY"),
57 + region: std::env::var("S3_TEST_REGION").unwrap_or_else(|_| "us-east-1".into()),
58 + }
59 + }
60 +
61 + async fn client() -> S3Client {
62 + S3Client::new(&config())
63 + .await
64 + .expect("building the client should not need the network")
65 + }
66 +
67 + /// A prefix no other run will use: the clock, the process, and a counter.
68 + ///
69 + /// All three are needed. The counter because two tests in one process start
70 + /// inside the same millisecond, and the pid because a mutation run has a dozen
71 + /// copies of this suite against one bucket at once (infra `1498fe2a`), where
72 + /// two runs sharing a prefix would delete each other's objects and report it as
73 + /// a killed mutant.
74 + fn prefix(what: &str) -> String {
75 + static SEQ: AtomicU32 = AtomicU32::new(0);
76 + let millis = SystemTime::now()
77 + .duration_since(UNIX_EPOCH)
78 + .expect("the clock is after 1970")
79 + .as_millis();
80 + let n = SEQ.fetch_add(1, Ordering::Relaxed);
81 + format!("run-{millis}-{}-{n}/{what}", std::process::id())
82 + }
83 +
84 + /// Bytes that are not all the same, so a wrong offset shows up as wrong content
85 + /// rather than as an identical-looking block.
86 + fn pattern(len: usize) -> Vec<u8> {
87 + (0..len).map(|i| (i % 251) as u8).collect()
88 + }
89 +
90 + /// Compare without printing megabytes on failure: say where they diverge.
91 + fn assert_same_bytes(expected: &[u8], actual: &[u8], what: &str) {
92 + assert_eq!(
93 + expected.len(),
94 + actual.len(),
95 + "{what}: length differs, expected {} bytes and got {}",
96 + expected.len(),
97 + actual.len()
98 + );
99 + if let Some(at) = expected.iter().zip(actual).position(|(a, b)| a != b) {
100 + panic!(
101 + "{what}: first difference at byte {at} of {}, expected {:#04x} and got {:#04x}",
102 + expected.len(),
103 + expected[at],
104 + actual[at]
105 + );
106 + }
107 + }
108 +
109 + /// Best-effort teardown. A failed cleanup must not turn a passing test red, and
110 + /// must not hide a failing one, so it reports and moves on.
111 + async fn cleanup(s3: &S3Client, prefix: &str) {
112 + if let Err(e) = s3.delete_prefix(prefix).await {
113 + eprintln!("cleanup of {prefix} failed, leaving objects behind: {e}");
114 + }
115 + }
116 +
117 + #[tokio::test]
118 + #[ignore = "live S3"]
119 + async fn an_object_round_trips_with_its_content_type() {
120 + let s3 = client().await;
121 + let key = prefix("round-trip/a.bin");
122 + let body = pattern(4096);
123 +
124 + s3.upload(
125 + &key,
126 + "application/octet-stream",
127 + body.clone(),
128 + Some("no-store"),
129 + )
130 + .await
131 + .expect("upload");
132 +
133 + let (got, content_type) = s3.download(&key).await.expect("download");
134 + assert_same_bytes(&body, &got, "round trip");
135 + assert_eq!(content_type, "application/octet-stream");
136 +
137 + assert!(s3.object_exists(&key).await.expect("exists"));
138 + assert_eq!(
139 + s3.object_size(&key).await.expect("size"),
140 + Some(body.len() as i64)
141 + );
142 +
143 + s3.delete(&key).await.expect("delete");
144 + assert!(
145 + !s3.object_exists(&key).await.expect("exists after delete"),
146 + "the object survived its own deletion"
147 + );
148 + assert_eq!(s3.object_size(&key).await.expect("size after delete"), None);
149 +
150 + cleanup(&s3, &key).await;
151 + }
152 +
153 + #[tokio::test]
154 + #[ignore = "live S3"]
155 + async fn download_head_reads_a_prefix_and_nothing_more() {
156 + let s3 = client().await;
157 + let key = prefix("head/a.bin");
158 + let body = pattern(10_000);
159 + s3.upload(&key, "application/octet-stream", body.clone(), None)
160 + .await
161 + .expect("upload");
162 +
163 + let head = s3.download_head(&key, 512).await.expect("ranged read");
164 + assert_same_bytes(&body[..512], &head, "ranged read");
165 +
166 + // Asking for more than the object holds is not an error, and the answer is
167 + // the object.
168 + let all = s3
169 + .download_head(&key, body.len() * 2)
170 + .await
171 + .expect("over-read");
172 + assert_same_bytes(&body, &all, "over-read");
173 +
174 + // Zero is answered without a request at all. It has to be: the range header
175 + // is built as `bytes=0-{len-1}`, which underflows at zero.
176 + assert!(
177 + s3.download_head(&key, 0)
178 + .await
179 + .expect("zero-length")
180 + .is_empty()
181 + );
182 +
183 + cleanup(&s3, &key).await;
184 + }
185 +
186 + #[tokio::test]
187 + #[ignore = "live S3"]
188 + async fn a_missing_object_is_absent_rather_than_an_error() {
189 + let s3 = client().await;
190 + let key = prefix("missing/never-written.bin");
191 + assert!(!s3.object_exists(&key).await.expect("exists"));
192 + assert_eq!(s3.object_size(&key).await.expect("size"), None);
193 + assert!(
194 + s3.download(&key).await.is_err(),
195 + "downloading nothing should fail rather than return empty"
196 + );
197 + }
198 +
199 + #[tokio::test]
200 + #[ignore = "live S3"]
201 + async fn a_copy_carries_the_bytes_not_just_a_status() {
202 + let s3 = client().await;
203 + let root = prefix("copy");
204 + let src = format!("{root}/src.bin");
205 + let dst = format!("{root}/dst.bin");
206 + let from = format!("{root}/from.bin");
207 + let body = pattern(65_536);
208 +
209 + s3.upload(&src, "application/octet-stream", body.clone(), None)
210 + .await
211 + .expect("upload");
212 +
213 + s3.copy_object(&src, &dst).await.expect("copy_object");
214 + let (got, _) = s3.download(&dst).await.expect("download copy");
215 + assert_same_bytes(&body, &got, "copy_object");
216 +
217 + // Same bucket, named explicitly: the cross-bucket form with the only bucket
218 + // these credentials can reach.
219 + s3.copy_object_from(s3.bucket(), &src, &from)
220 + .await
221 + .expect("copy_object_from");
222 + let (got, _) = s3.download(&from).await.expect("download copy_from");
223 + assert_same_bytes(&body, &got, "copy_object_from");
224 +
225 + cleanup(&s3, &root).await;
226 + }
227 +
228 + #[tokio::test]
229 + #[ignore = "live S3"]
230 + async fn deleting_a_batch_reports_no_failures_and_removes_every_key() {
231 + let s3 = client().await;
232 + let root = prefix("batch");
233 + let keys: Vec<String> = (0..5).map(|i| format!("{root}/{i}.bin")).collect();
234 + for key in &keys {
235 + s3.upload(key, "application/octet-stream", pattern(32), None)
236 + .await
237 + .expect("upload");
238 + }
239 + // A key that was never written, because a batch in the wild has them and S3
240 + // treats deleting an absent key as success.
241 + let mut with_absent = keys.clone();
242 + with_absent.push(format!("{root}/never-written.bin"));
243 +
244 + let failures = s3
245 + .delete_objects(&with_absent)
246 + .await
247 + .expect("delete_objects");
248 + assert!(failures.is_empty(), "delete_objects reported {failures:?}");
249 + for key in &keys {
250 + assert!(
251 + !s3.object_exists(key).await.expect("exists"),
252 + "{key} survived the batch delete"
253 + );
254 + }
255 +
256 + cleanup(&s3, &root).await;
257 + }
258 +
259 + #[tokio::test]
260 + #[ignore = "live S3"]
261 + async fn delete_prefix_stops_at_the_prefix() {
262 + let s3 = client().await;
263 + let root = prefix("wipe");
264 + let doomed = format!("{root}/doomed");
265 + let kept = format!("{root}/kept");
266 + for i in 0..3 {
267 + s3.upload(
268 + &format!("{doomed}/{i}.bin"),
269 + "application/octet-stream",
270 + pattern(16),
271 + None,
272 + )
273 + .await
274 + .expect("upload");
275 + }
276 + s3.upload(
277 + &format!("{kept}/survivor.bin"),
278 + "application/octet-stream",
279 + pattern(16),
280 + None,
281 + )
282 + .await
283 + .expect("upload");
284 +
285 + s3.delete_prefix(&doomed).await.expect("delete_prefix");
286 +
287 + for i in 0..3 {
288 + assert!(
289 + !s3.object_exists(&format!("{doomed}/{i}.bin"))
290 + .await
291 + .expect("exists"),
292 + "object {i} survived delete_prefix"
293 + );
294 + }
295 + assert!(
296 + s3.object_exists(&format!("{kept}/survivor.bin"))
297 + .await
298 + .expect("exists"),
299 + "delete_prefix reached past its prefix, which is how a wipe becomes an incident"
300 + );
301 +
302 + cleanup(&s3, &root).await;
303 + }
304 +
305 + #[tokio::test]
306 + #[ignore = "live S3"]
307 + async fn a_multipart_upload_reads_back_byte_identical() {
308 + let s3 = client().await;
309 + let key = prefix("multipart/big.bin");
310 + // Three parts at the 5 MiB floor, the last one deliberately short: a driver
311 + // that gets the final part's length wrong passes on an exact multiple.
312 + let part_size = 5 * 1024 * 1024;
313 + let body = pattern(part_size * 2 + 1_234_567);
314 +
315 + let file = std::env::temp_dir().join(format!("s3-live-{}.bin", std::process::id()));
316 + std::fs::write(&file, &body).expect("writing the source file");
317 +
318 + let uploaded = s3
319 + .upload_multipart(&key, "application/octet-stream", &file, Some(part_size))
320 + .await;
321 + std::fs::remove_file(&file).ok();
322 + uploaded.expect("upload_multipart");
323 +
324 + assert_eq!(
325 + s3.object_size(&key).await.expect("size"),
326 + Some(body.len() as i64),
327 + "the assembled object is the wrong length"
328 + );
329 + let (got, _) = s3.download(&key).await.expect("download");
330 + assert_same_bytes(&body, &got, "multipart round trip");
331 +
332 + cleanup(&s3, &key).await;
333 + }
334 +
335 + #[tokio::test]
336 + #[ignore = "live S3"]
337 + async fn a_multipart_copy_preserves_every_byte() {
338 + let s3 = client().await;
339 + let root = prefix("multipart-copy");
340 + let src = format!("{root}/src.bin");
341 + let dst = format!("{root}/dst.bin");
342 + let part_size = 5 * 1024 * 1024;
343 + let body = pattern(part_size * 2 + 999_983);
344 +
345 + let file = std::env::temp_dir().join(format!("s3-live-copy-{}.bin", std::process::id()));
346 + std::fs::write(&file, &body).expect("writing the source file");
347 + let uploaded = s3
348 + .upload_multipart(&src, "application/octet-stream", &file, Some(part_size))
349 + .await;
350 + std::fs::remove_file(&file).ok();
351 + uploaded.expect("upload_multipart");
352 +
353 + s3.copy_object_multipart(
354 + s3.bucket(),
355 + &src,
356 + &dst,
357 + "application/octet-stream",
358 + body.len() as u64,
359 + Some(part_size),
360 + )
361 + .await
362 + .expect("copy_object_multipart");
363 +
364 + let (got, _) = s3.download(&dst).await.expect("download");
365 + // THE ASSERTION THIS FILE WAS WRITTEN FOR. A copy driver whose part offsets
366 + // are off by one part duplicates or drops a range, the object still
367 + // completes, and nothing but the bytes says so.
368 + assert_same_bytes(&body, &got, "multipart copy");
369 +
370 + cleanup(&s3, &root).await;
371 + }
372 +
373 + #[tokio::test]
374 + #[ignore = "live S3"]
375 + async fn an_aborted_upload_leaves_nothing_behind() {
376 + let s3 = client().await;
377 + let key = prefix("abort/pending.bin");
378 +
379 + let upload_id = s3
380 + .create_multipart_upload(&key, "application/octet-stream")
381 + .await
382 + .expect("create_multipart_upload");
383 + let pending = s3
384 + .list_multipart_uploads_for_key(&key)
385 + .await
386 + .expect("list_multipart_uploads_for_key");
387 + assert!(
388 + pending.contains(&upload_id),
389 + "the upload we just created is not listed as pending: {pending:?}"
390 + );
391 +
392 + s3.abort_multipart_upload(&key, &upload_id)
393 + .await
394 + .expect("abort_multipart_upload");
395 + let after = s3
396 + .list_multipart_uploads_for_key(&key)
397 + .await
398 + .expect("list after abort");
399 + assert!(
400 + !after.contains(&upload_id),
401 + "the aborted upload is still pending, which is a bill nobody sees: {after:?}"
402 + );
403 + assert!(
404 + !s3.object_exists(&key).await.expect("exists"),
405 + "an aborted upload produced an object"
406 + );
407 + }
408 +
409 + #[tokio::test]
410 + #[ignore = "live S3"]
411 + async fn a_presigned_url_actually_fetches_and_actually_uploads() {
412 + let s3 = client().await;
413 + let root = prefix("presign");
414 + let download_key = format!("{root}/download.bin");
415 + let upload_key = format!("{root}/upload.bin");
416 + let body = pattern(2048);
417 +
418 + s3.upload(
419 + &download_key,
420 + "application/octet-stream",
421 + body.clone(),
422 + None,
423 + )
424 + .await
425 + .expect("upload");
426 +
427 + let url = s3
428 + .presign_download(&download_key, 300)
429 + .await
430 + .expect("presign_download");
431 + let fetched = http_get(&url).expect("fetching the presigned URL");
432 + assert_same_bytes(&body, &fetched, "presigned download");
433 +
434 + let put_url = s3
435 + .presign_upload(
436 + &upload_key,
437 + "application/octet-stream",
438 + 300,
439 + None,
440 + Some(body.len() as i64),
441 + )
442 + .await
443 + .expect("presign_upload");
444 + http_put(&put_url, "application/octet-stream", &body).expect("PUT to the presigned URL");
445 + let (got, _) = s3
446 + .download(&upload_key)
447 + .await
448 + .expect("download what was PUT");
449 + assert_same_bytes(&body, &got, "presigned upload");
450 +
451 + cleanup(&s3, &root).await;
452 + }
453 +
454 + #[tokio::test]
455 + #[ignore = "live S3"]
456 + async fn connectivity_answers_for_the_configured_bucket() {
457 + let s3 = client().await;
458 + s3.check_connectivity().await.expect("check_connectivity");
459 +
460 + let mut wrong = config();
461 + wrong.bucket = format!("{}-does-not-exist", wrong.bucket);
462 + let s3 = S3Client::new(&wrong).await.expect("client");
463 + assert!(
464 + s3.check_connectivity().await.is_err(),
465 + "connectivity passed against a bucket that does not exist, so it is not checking the bucket"
466 + );
467 + }
468 +
469 + // ---------------------------------------------------------------------------
470 + // A minimal HTTP client, because a presigned URL is only worth anything if
471 + // something outside this crate can use it.
472 + //
473 + // Raw TCP rather than a dependency: the tier's endpoint is MinIO on localhost
474 + // over plain HTTP, and adding an HTTP stack to dev-dependencies to issue two
475 + // requests would pull a TLS backend into a crate that deliberately pins its own
476 + // (see Cargo.toml on `rustls-ring`). If the tier ever points at an https
477 + // endpoint, these two helpers are what has to change, and they will fail loudly
478 + // rather than quietly skip.
479 + // ---------------------------------------------------------------------------
480 +
481 + fn split_url(url: &str) -> (String, String) {
482 + let rest = url
483 + .strip_prefix("http://")
484 + .unwrap_or_else(|| panic!("this helper speaks plain HTTP only, got: {url}"));
485 + match rest.split_once('/') {
486 + Some((host, path)) => (host.to_string(), format!("/{path}")),
487 + None => (rest.to_string(), "/".to_string()),
488 + }
489 + }
490 +
491 + fn read_response(mut stream: TcpStream) -> Result<Vec<u8>, String> {
492 + let mut raw = Vec::new();
493 + stream
494 + .read_to_end(&mut raw)
495 + .map_err(|e| format!("reading the response: {e}"))?;
496 + let split = raw
497 + .windows(4)
498 + .position(|w| w == b"\r\n\r\n")
499 + .ok_or_else(|| "no header terminator in the response".to_string())?;
500 + let headers = String::from_utf8_lossy(&raw[..split]).to_string();
Lines truncated