Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 62 additions & 1 deletion apps/web/app/api/upload/[...route]/multipart.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { serverEnv } from "@cap/env";
import { userIsPro } from "@cap/utils";
import {
Database,
MAX_UPLOAD_BYTES,
makeCurrentUserLayer,
provideOptionalAuth,
Storage,
Expand Down Expand Up @@ -290,7 +291,10 @@ app.post(
z.object({
partNumber: z.number(),
etag: z.string(),
size: z.number(),
// A non-negative integer (bytes) so neither a negative nor a
// fractional size can drag the summed total below the cap and
// bypass the upload-size limit.
size: z.number().int().nonnegative(),
}),
),
durationInSecs: stringOrNumberOptional,
Expand Down Expand Up @@ -399,6 +403,63 @@ app.post(
}
}

// Server-side backstop for the maximum upload size. Presigned POST URLs
// enforce a content-length-range policy, but presigned PUT part URLs
// cannot enforce a total size, so reject an oversized assembled upload
// here before persisting (and before paying to assemble it). Part sizes

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth calling out: this guard trusts part.size from the request body. A malicious client can under-report sizes and still complete an oversized MPU (S3 only cares about PartNumber/ETag). If the intent is a hard storage cap, you probably need to sum authoritative sizes from S3 (ListParts) or enforce Content-Length on the presigned part PUTs.

// are client-reported, so this raises the bar rather than enforcing
// authoritatively.
let totalUploadSize = 0;
for (const part of parts) {
totalUploadSize += part.size;
Comment thread
richiemcilroy marked this conversation as resolved.
if (totalUploadSize > MAX_UPLOAD_BYTES) break;
}
if (totalUploadSize > MAX_UPLOAD_BYTES) {
// Avoid leaving the parts as incomplete-MPU storage and a stale
// videoUploads row, mirroring the free-plan rejection cleanup. Each
// step is caught independently so a failed abort doesn't skip the DB
// cleanup (and vice versa). The 413 stands regardless of either.
yield* Effect.gen(function* () {
Comment thread
richiemcilroy marked this conversation as resolved.
const [bucket] = yield* Storage.getAccessForVideo(video);
yield* bucket.multipart
.abort(fileKey, uploadId)
.pipe(
Effect.catchAll((error) =>
Effect.logError(
"Failed to abort rejected oversized multipart upload",
error,
),
),
);
yield* db
.use((db) =>
db
.delete(Db.videoUploads)
.where(eq(Db.videoUploads.videoId, videoId)),
)
.pipe(
Effect.catchAll((error) =>
Effect.logError(
"Failed to delete videoUploads row for rejected upload",
error,
),
),
);
}).pipe(
Effect.catchAll((error) =>
Effect.logError(
"Failed to clean up rejected oversized multipart upload",
error,
),
),
);

c.status(413);
return c.text(
"Upload exceeds the maximum allowed size and cannot be completed.",
);
}

return yield* Effect.gen(function* () {
const [bucket] = yield* Storage.getAccessForVideo(video);

Expand Down
49 changes: 45 additions & 4 deletions packages/web-backend/src/S3Buckets/S3BucketAccess.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ import { S3BucketClientProvider } from "./S3BucketClientProvider.ts";
const DEFAULT_PRESIGNED_GET_EXPIRES_SECONDS = 3600;
const DEFAULT_PRESIGNED_PUT_EXPIRES_SECONDS = 3600;

// Upper bound on a single upload to prevent unbounded storage abuse. Generous
// on purpose so legitimate long/high-bitrate recordings are never blocked;
// tune here if the product ever needs a larger ceiling.
export const MAX_UPLOAD_BYTES = 100 * 1024 * 1024 * 1024; // 100 GiB

type NodeReadableWebStream = Parameters<typeof Readable.fromWeb>[0];

const wrapS3Promise = <T>(
Expand Down Expand Up @@ -266,13 +271,49 @@ export const createS3BucketAccess = Effect.gen(function* () {
) =>
wrapS3Promise(
provider.getPublic.pipe(
Effect.map((client) =>
createPresignedPost(client, {
Effect.map((client) => {
// Enforce an upper bound on the uploaded object size. The POST
// policy rejects the upload at S3 if the body exceeds this,
// closing the unbounded-storage hole for presigned POSTs. We emit
// exactly one content-length-range whose max never exceeds
// MAX_UPLOAD_BYTES, even if a caller supplied a looser one, so a
// caller can tighten but never raise/disable the cap.
const callerConditions = args.Conditions ?? [];
Comment thread
richiemcilroy marked this conversation as resolved.
const isLengthRange = (condition: unknown): boolean =>
Array.isArray(condition) &&
condition[0] === "content-length-range";
const callerRange = callerConditions.find(isLengthRange) as
| [unknown, unknown, unknown]
| undefined;
const callerMin =
callerRange &&
typeof callerRange[1] === "number" &&
Number.isFinite(callerRange[1])
? Math.max(0, callerRange[1])
: 0;
const callerMax =
callerRange &&
typeof callerRange[2] === "number" &&
Number.isFinite(callerRange[2])
? Math.max(0, callerRange[2])
: MAX_UPLOAD_BYTES;
Comment thread
richiemcilroy marked this conversation as resolved.
const otherConditions = callerConditions.filter(
(condition) => !isLengthRange(condition),
);
return createPresignedPost(client, {
...args,
Conditions: [
[
"content-length-range",
callerMin,
Math.min(callerMax, MAX_UPLOAD_BYTES),
],
...otherConditions,
],
Comment on lines +288 to +312

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One edge case: if the caller supplies a content-length-range where min exceeds MAX_UPLOAD_BYTES (or min > max), we’ll generate an invalid policy (min > max). Might be worth clamping + normalizing so we always emit a valid range.

Suggested change
const callerMin =
callerRange &&
typeof callerRange[1] === "number" &&
Number.isFinite(callerRange[1])
? Math.max(0, callerRange[1])
: 0;
const callerMax =
callerRange &&
typeof callerRange[2] === "number" &&
Number.isFinite(callerRange[2])
? Math.max(0, callerRange[2])
: MAX_UPLOAD_BYTES;
const otherConditions = callerConditions.filter(
(condition) => !isLengthRange(condition),
);
return createPresignedPost(client, {
...args,
Conditions: [
[
"content-length-range",
callerMin,
Math.min(callerMax, MAX_UPLOAD_BYTES),
],
...otherConditions,
],
const callerMin =
callerRange &&
typeof callerRange[1] === "number" &&
Number.isFinite(callerRange[1])
? Math.max(0, callerRange[1])
: 0;
const callerMax =
callerRange &&
typeof callerRange[2] === "number" &&
Number.isFinite(callerRange[2])
? Math.max(0, callerRange[2])
: MAX_UPLOAD_BYTES;
const minBytes = Math.min(callerMin, MAX_UPLOAD_BYTES);
const maxBytes = Math.min(Math.max(callerMax, minBytes), MAX_UPLOAD_BYTES);
const otherConditions = callerConditions.filter(
(condition) => !isLengthRange(condition),
);
return createPresignedPost(client, {
...args,
Conditions: [
["content-length-range", minBytes, maxBytes],
...otherConditions,
],
Bucket: provider.bucket,
Key: key,
});

Bucket: provider.bucket,
Key: key,
}),
),
});
}),
),
),
multipart: {
Expand Down
1 change: 1 addition & 0 deletions packages/web-backend/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ export { Organisations } from "./Organisations/index.ts";
export { OrganisationsPolicy } from "./Organisations/OrganisationsPolicy.ts";
export * from "./Rpcs.ts";
export { S3Buckets } from "./S3Buckets/index.ts";
export { MAX_UPLOAD_BYTES } from "./S3Buckets/S3BucketAccess.ts";
export { Spaces } from "./Spaces/index.ts";
export { SpacesPolicy } from "./Spaces/SpacesPolicy.ts";
export * from "./Storage/GoogleDrive.ts";
Expand Down
Loading