-
Notifications
You must be signed in to change notification settings - Fork 3
Expand file tree
/
Copy pathSPEC.html
More file actions
1746 lines (1746 loc) · 86.3 KB
/
Copy pathSPEC.html
File metadata and controls
1746 lines (1746 loc) · 86.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Box Archive Format Specification</title>
<meta name="authors" content="Brendan Molloy">
<link rel="stylesheet" href="spec.css">
</head>
<body>
<div>
<p>Box Archive Format Specification</p>
<p><strong>Version:</strong> 0.2.0<br><strong>Status:</strong> Draft</p>
</div>
<nav role="doc-toc">
<h2>Contents</h2>
<ol style="list-style-type: none">
<li>
<div><span style="text-decoration: underline"><a href="#loc-1"><span class="prefix">1</span> Introduction</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-2"><span class="prefix">1.1</span> Design Goals</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-3"><span class="prefix">1.2</span> File Extension</a></span></li>
</ol>
</li>
<li><span style="text-decoration: underline"><a href="#loc-4"><span class="prefix">2</span> Terminology</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-5"><span class="prefix">3</span> File Structure Overview</a></span></li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-6"><span class="prefix">4</span> Data Types</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-7"><span class="prefix">4.1</span> Fixed-Size Integers</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-8"><span class="prefix">4.2</span> Vu64 Encoding (FastVint)</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-9"><span class="prefix">4.3</span> String</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-10"><span class="prefix">4.4</span> Vec<T></a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-11"><span class="prefix">4.5</span> Vec<u8> (Byte Array)</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-12"><span class="prefix">4.6</span> RecordIndex</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-13"><span class="prefix">4.7</span> AttrMap</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-14"><span class="prefix">5</span> Header</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-15"><span class="prefix">5.1</span> Header Structure</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-16"><span class="prefix">5.2</span> Magic Bytes</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-17"><span class="prefix">5.3</span> Version</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-18"><span class="prefix">5.4</span> Flags</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-19"><span class="prefix">5.5</span> Alignment</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-20"><span class="prefix">5.6</span> Reserved Fields</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-21"><span class="prefix">5.7</span> Binary Layout</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-22"><span class="prefix">6</span> Data Section</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-23"><span class="prefix">6.1</span> Layout</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-24"><span class="prefix">6.2</span> Alignment Padding</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-25"><span class="prefix">7</span> Trailer (Metadata)</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-26"><span class="prefix">7.1</span> BoxMetadata Structure</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-27"><span class="prefix">7.2</span> Attribute Key Table (attr_keys)</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-28"><span class="prefix">7.3</span> Attribute Value Types</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-29"><span class="prefix">7.4</span> Archive Attributes (attrs)</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-30"><span class="prefix">7.5</span> Dictionary</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-31"><span class="prefix">8</span> Record Types</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-32"><span class="prefix">8.1</span> Type/Compression Byte</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-33"><span class="prefix">8.2</span> Directory Record (Type 0x01)</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-34"><span class="prefix">8.3</span> File Record (Type 0x02)</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-35"><span class="prefix">8.4</span> Symlink Record (Type 0x03)</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-36"><span class="prefix">8.5</span> External Symlink Record (Type 0x0B)</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-37"><span class="prefix">8.6</span> Chunked File Record (Type 0x0A)</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-38"><span class="prefix">9</span> Path Encoding</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-39"><span class="prefix">9.1</span> BoxPath Format</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-40"><span class="prefix">9.2</span> Separator Choice</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-41"><span class="prefix">9.3</span> Platform Conversion</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-42"><span class="prefix">9.4</span> Unicode Normalization</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-43"><span class="prefix">10</span> Compression</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-44"><span class="prefix">10.1</span> Supported Methods</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-45"><span class="prefix">10.2</span> Zstd Configuration</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-46"><span class="prefix">10.3</span> XZ Configuration</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-47"><span class="prefix">10.4</span> Stored (Uncompressed)</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-48"><span class="prefix">10.5</span> Dictionary Usage</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-49"><span class="prefix">11</span> Checksums</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-50"><span class="prefix">11.1</span> Supported Algorithms</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-51"><span class="prefix">11.2</span> Checksum Storage</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-52"><span class="prefix">11.3</span> Verification</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-53"><span class="prefix">12</span> Standard Attributes</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-54"><span class="prefix">12.1</span> Timestamps</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-55"><span class="prefix">12.2</span> Sub-Minute Precision</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-56"><span class="prefix">12.3</span> Unix Permissions</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-57"><span class="prefix">12.4</span> Attribute Encoding</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-58"><span class="prefix">13</span> Implementation Guidance</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-59"><span class="prefix">13.1</span> Memory Mapping</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-60"><span class="prefix">13.2</span> Streaming Creation</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-61"><span class="prefix">13.3</span> Large File Handling</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-62"><span class="prefix">13.4</span> Error Recovery</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-63"><span class="prefix">14</span> Path FST</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-64"><span class="prefix">14.1</span> Overview</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-65"><span class="prefix">14.2</span> FST Layout</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-66"><span class="prefix">14.3</span> FST Header</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-67"><span class="prefix">14.4</span> Node Index</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-68"><span class="prefix">14.5</span> Hot Section</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-69"><span class="prefix">14.6</span> Cold Section</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-70"><span class="prefix">14.7</span> Path Encoding in FST</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-71"><span class="prefix">14.8</span> Output Value Accumulation</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-72"><span class="prefix">14.9</span> FST Implementation Notes</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-73"><span class="prefix">14.10</span> FST Constants</a></span></li>
</ol>
</li>
<li>
<div><span style="text-decoration: underline"><a href="#loc-74"><span class="prefix">15</span> Block FST</a></span></div>
<ol style="list-style-type: none">
<li><span style="text-decoration: underline"><a href="#loc-75"><span class="prefix">15.1</span> Block FST Overview</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-76"><span class="prefix">15.2</span> Block FST Format</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-77"><span class="prefix">15.3</span> Block FST Key Format</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-78"><span class="prefix">15.4</span> Block FST Value Format</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-79"><span class="prefix">15.5</span> Block Lookup</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-80"><span class="prefix">15.6</span> Block FST Build Requirements</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-81"><span class="prefix">15.7</span> Streaming Decompression</a></span></li>
<li><span style="text-decoration: underline"><a href="#loc-82"><span class="prefix">15.8</span> Block FST Constants</a></span></li>
</ol>
</li>
</ol>
</nav>
<h2 id="loc-1">1 Introduction</h2>
<p>The Box format is an open archive format designed for storing files with support for compression, checksums, extended attributes, and memory-mapped access. It provides platform-independent path encoding and efficient metadata storage through string interning.</p>
<h3 id="loc-2">1.1 Design Goals</h3>
<ul>
<li><strong>Platform Independence:</strong> Paths are encoded in a platform-neutral format that can be safely represented on all major operating systems.</li>
<li><strong>Compression:</strong> Multiple compression algorithms are supported, selectable per-file.</li>
<li><strong>Integrity:</strong> Optional Blake3 checksums for content verification.</li>
<li><strong>Efficiency:</strong> String interning for attribute keys reduces metadata size. Memory mapping enables zero-copy file access for uncompressed files. Block-level compression enables efficient random access for large compressed files.</li>
<li><strong>Extensibility:</strong> The attribute system allows arbitrary metadata without format changes.</li>
</ul>
<h3 id="loc-3">1.2 File Extension</h3>
<p>Box archives SHOULD use the <span style="display: inline-block"><code>.box</code></span> file extension.</p>
<h2 id="loc-4">2 Terminology</h2>
<p>The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in <span style="text-decoration: underline"><a href="https://www.rfc-editor.org/rfc/rfc2119">RFC 2119</a></span>.</p>
<p><strong>Archive:</strong> A Box format file containing zero or more entries.</p>
<p><strong>Entry:</strong> A file, directory, or symbolic link stored in the archive.</p>
<p><strong>Record:</strong> The metadata structure describing an entry.</p>
<p><strong>Trailer:</strong> The metadata section at the end of the archive containing all records and attributes.</p>
<p><strong>Chunked File:</strong> A file stored with block-level compression, enabling random access without decompressing the entire file.</p>
<h2 id="loc-5">3 File Structure Overview</h2>
<p>A Box archive consists of five sections laid out sequentially:</p>
<figure>
<table>
<tr>
<td>Offset 0x00 →</td>
<td><strong>Header</strong></td>
<td>32 bytes, fixed size</td>
</tr>
<tr>
<td>Offset 0x20 →</td>
<td><strong>Data Section</strong></td>
<td>Variable size</td>
</tr>
<tr>
<td>header.trailer →</td>
<td><strong>Trailer</strong></td>
<td>Variable size (BoxMetadata)</td>
</tr>
<tr>
<td></td>
<td><strong>Path FST</strong></td>
<td>Variable size</td>
</tr>
<tr>
<td>EOF →</td>
<td><strong>Block FST</strong></td>
<td>Variable size (optional)</td>
</tr>
</table>
<figcaption>Table 1: Box archive structure</figcaption>
</figure>
<p>The Header is always 32 bytes and located at offset 0. The Header contains a pointer to the Trailer, which stores record metadata. File content data is stored between the Header and Trailer. The Path FST follows the Trailer and provides path-to-record mapping (see <span style="text-decoration: underline"><a href="#loc-63">Section 14</a></span>). The Block FST, if present, follows the Path FST and provides logical-to-physical offset mapping for chunked files (see <span style="text-decoration: underline"><a href="#loc-74">Section 15</a></span>). Both FSTs are prefixed with a u64 length (little-endian), enabling efficient seeking past the Path FST to detect the Block FST’s presence.</p>
<p>All multi-byte integers in the Box format are stored in <strong>little-endian</strong> byte order unless otherwise specified.</p>
<h2 id="loc-6">4 Data Types</h2>
<h3 id="loc-7">4.1 Fixed-Size Integers</h3>
<table>
<thead>
<tr>
<th><strong>Type</strong></th>
<th><strong>Size</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td><span style="display: inline-block"><code>u8</code></span></td>
<td>1 byte</td>
<td>Unsigned 8-bit integer</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>u32</code></span></td>
<td>4 bytes</td>
<td>Unsigned 32-bit integer, little-endian</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>u64</code></span></td>
<td>8 bytes</td>
<td>Unsigned 64-bit integer, little-endian</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>i64</code></span></td>
<td>8 bytes</td>
<td>Signed 64-bit integer, little-endian</td>
</tr>
</tbody>
</table>
<h3 id="loc-8">4.2 Vu64 Encoding (FastVint)</h3>
<p>Variable-size unsigned 64-bit integers (Vu64) are encoded using FastVint, a prefix-based variable-length encoding where the number of leading zeros in the first byte determines the total byte count.</p>
<p><strong>Length Determination:</strong></p>
<p>The number of leading zero bits in the first byte, plus one, gives the total byte count:</p>
<div>
<pre><code>1xxx_xxxx → 1 byte (7 data bits)<br>01xx_xxxx → 2 bytes (14 data bits: 6 + 8)<br>001x_xxxx → 3 bytes (21 data bits: 5 + 16)<br>0001_xxxx → 4 bytes (28 data bits: 4 + 24)<br>0000_1xxx → 5 bytes (35 data bits: 3 + 32)<br>0000_01xx → 6 bytes (42 data bits: 2 + 40)<br>0000_001x → 7 bytes (49 data bits: 1 + 48)<br>0000_0001 → 8 bytes (56 data bits: 0 + 56)<br>0000_0000 → 9 bytes (64 data bits: 0 + 64)</code></pre>
</div>
<p><strong>Offset-Based Encoding:</strong></p>
<p>FastVint uses offset-based encoding for denser packing. Each length tier has a base offset:</p>
<table>
<thead>
<tr>
<th><strong>Bytes</strong></th>
<th><strong>Value Range</strong></th>
<th><strong>Offset</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>0 – 127</td>
<td>0</td>
</tr>
<tr>
<td>2</td>
<td>128 – 16,511</td>
<td>128</td>
</tr>
<tr>
<td>3</td>
<td>16,512 – 2,113,663</td>
<td>16,512</td>
</tr>
<tr>
<td>4</td>
<td>2,113,664 – 270,549,119</td>
<td>2,113,664</td>
</tr>
<tr>
<td>5+</td>
<td>…</td>
<td>(continues exponentially)</td>
</tr>
<tr>
<td>9</td>
<td>(full u64 range)</td>
<td>72,624,976,668,147,840</td>
</tr>
</tbody>
</table>
<p><strong>Data Byte Order:</strong></p>
<p>After the prefix byte, remaining data bytes are stored in <strong>big-endian</strong> order.</p>
<p><strong>Key Properties:</strong></p>
<ul>
<li>Length is known from the first byte alone (no continuation scanning required)</li>
<li>Maximum 9 bytes for <span style="display: inline-block"><code>u64</code></span> values</li>
<li>This encoding does NOT enforce minimal representations; the same value MAY have multiple valid encodings</li>
<li>Decoders MUST accept all valid encodings of a value, including non-minimal representations</li>
</ul>
<h3 id="loc-9">4.3 String</h3>
<p>Strings are encoded as:</p>
<div>
<pre><code>[Vu64: byte length]<br>[UTF-8 bytes]</code></pre>
</div>
<p>All strings MUST be valid UTF-8. Implementations MUST reject invalid UTF-8 sequences.</p>
<h3 id="loc-10">4.4 Vec<T></h3>
<p>Vectors (arrays) are encoded as:</p>
<div>
<pre><code>[Vu64: element count]<br>[T encoding] * count</code></pre>
</div>
<p>The following vector types are used in the format:</p>
<table>
<thead>
<tr>
<th><strong>Type</strong></th>
<th><strong>Usage</strong></th>
<th><strong>Element Encoding</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td><span style="display: inline-block"><code>Vec<Record></code></span></td>
<td><span style="display: inline-block"><code>BoxMetadata.records</code></span></td>
<td>Record (see <span style="text-decoration: underline"><a href="#loc-31">Section 8</a></span>)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>Vec<AttrKey></code></span></td>
<td><span style="display: inline-block"><code>attr_keys</code></span></td>
<td>Type tag + String (see <span style="text-decoration: underline"><a href="#loc-27">Section 7.2</a></span>)</td>
</tr>
</tbody>
</table>
<h3 id="loc-11">4.5 Vec<u8> (Byte Array)</h3>
<p>Byte arrays are encoded identically to strings:</p>
<div>
<pre><code>[Vu64: byte length]<br>[raw bytes]</code></pre>
</div>
<h3 id="loc-12">4.6 RecordIndex</h3>
<p>A RecordIndex is a 1-based index into the records array, encoded as Vu64:</p>
<div>
<pre><code>[Vu64: index value]</code></pre>
</div>
<p>Index values MUST be non-zero. An index value of <span style="display: inline-block"><code>n</code></span> refers to <span style="display: inline-block"><code>records[n-1]</code></span>.</p>
<h3 id="loc-13">4.7 AttrMap</h3>
<p>Attribute maps are encoded as:</p>
<div>
<pre><code>[u64: byte count of remaining section (not including this field)]<br>[Vu64: entry count]<br>For each entry:<br> [Vu64: key index into attr_keys]<br> [Vec<u8>: value]</code></pre>
</div>
<p>The leading byte count allows implementations to skip the entire attribute map without parsing individual entries.</p>
<h2 id="loc-14">5 Header</h2>
<p>The Header is located at byte offset 0 and is exactly 32 bytes.</p>
<h3 id="loc-15">5.1 Header Structure</h3>
<table>
<thead>
<tr>
<th><strong>Offset</strong></th>
<th><strong>Size</strong></th>
<th><strong>Field</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td><span style="display: inline-block"><code>0x00</code></span></td>
<td>4</td>
<td><span style="display: inline-block"><code>magic</code></span></td>
<td>Magic bytes: <span style="display: inline-block"><code>0xFF 0x42 0x4F 0x58</code></span> (<span style="display: inline-block"><code>\xFFBOX</code></span>)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x04</code></span></td>
<td>1</td>
<td><span style="display: inline-block"><code>version</code></span></td>
<td>Format version number</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x05</code></span></td>
<td>1</td>
<td><span style="display: inline-block"><code>flags</code></span></td>
<td>Feature flags (see <span style="text-decoration: underline"><a href="#loc-18">Section 5.4</a></span>)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x06</code></span></td>
<td>2</td>
<td><span style="display: inline-block"><code>reserved1</code></span></td>
<td>Reserved bytes</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x08</code></span></td>
<td>4</td>
<td><span style="display: inline-block"><code>alignment</code></span></td>
<td>Data alignment boundary (<span style="display: inline-block"><code>u32</code></span>)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x0C</code></span></td>
<td>4</td>
<td><span style="display: inline-block"><code>reserved2</code></span></td>
<td>Reserved bytes</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x10</code></span></td>
<td>8</td>
<td><span style="display: inline-block"><code>trailer</code></span></td>
<td>Byte offset to trailer (<span style="display: inline-block"><code>u64</code></span>)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x18</code></span></td>
<td>8</td>
<td><span style="display: inline-block"><code>reserved3</code></span></td>
<td>Reserved bytes</td>
</tr>
</tbody>
</table>
<p><strong>Total Size:</strong> 32 bytes</p>
<h3 id="loc-16">5.2 Magic Bytes</h3>
<p>The magic bytes MUST be exactly <span style="display: inline-block"><code>0xFF 0x42 0x4F 0x58</code></span> (the byte <span style="display: inline-block"><code>0xFF</code></span> followed by ASCII <span style="display: inline-block"><code>BOX</code></span>).</p>
<p>The leading <span style="display: inline-block"><code>0xFF</code></span> byte is invalid UTF-8, causing any attempt to parse the file as a text string to fail immediately at byte 0.</p>
<h3 id="loc-17">5.3 Version</h3>
<p>The current format version is <span style="display: inline-block"><code>0x01</code></span> (1). Implementations SHOULD reject archives with unknown major versions.</p>
<h3 id="loc-18">5.4 Flags</h3>
<p>The flags byte is a bitfield:</p>
<table>
<thead>
<tr>
<th><strong>Bit</strong></th>
<th><strong>Name</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td><span style="display: inline-block"><code>EXTERNAL_SYMLINKS</code></span></td>
<td>Archive contains external symlinks (see <span style="text-decoration: underline"><a href="#loc-36">Section 8.5</a></span>)</td>
</tr>
<tr>
<td>1</td>
<td><span style="display: inline-block"><code>ALLOW_ESCAPES</code></span></td>
<td>Archive paths may contain <span style="display: inline-block"><code>..</code></span> components</td>
</tr>
<tr>
<td>2-7</td>
<td>Reserved</td>
<td>Must be 0</td>
</tr>
</tbody>
</table>
<h3 id="loc-19">5.5 Alignment</h3>
<p>The <span style="display: inline-block"><code>alignment</code></span> field specifies the byte boundary to which file data is aligned. Valid values are powers of 2 from 1 to 65536. A value of 0 indicates no alignment (equivalent to 1).</p>
<p>When alignment is specified, padding bytes (value <span style="display: inline-block"><code>0x00</code></span>) are inserted before file data to ensure each file’s data starts at an offset that is a multiple of the alignment value.</p>
<p>Common alignment values:</p>
<ul>
<li><span style="display: inline-block"><code>0</code></span> or <span style="display: inline-block"><code>1</code></span>: No alignment (most compact)</li>
<li><span style="display: inline-block"><code>4096</code></span>: Page alignment (optimal for memory mapping)</li>
<li><span style="display: inline-block"><code>512</code></span>: Sector alignment</li>
</ul>
<h3 id="loc-20">5.6 Reserved Fields</h3>
<p>All reserved fields (<span style="display: inline-block"><code>reserved1</code></span>, <span style="display: inline-block"><code>reserved2</code></span>, <span style="display: inline-block"><code>reserved3</code></span>) MUST be set to zero when writing and MUST be ignored when reading to allow for future format extensions.</p>
<h3 id="loc-21">5.7 Binary Layout</h3>
<figure>
<table>
<tr>
<td><strong>Off</strong></td>
<td><strong>00</strong></td>
<td><strong>01</strong></td>
<td><strong>02</strong></td>
<td><strong>03</strong></td>
<td><strong>04</strong></td>
<td><strong>05</strong></td>
<td><strong>06</strong></td>
<td><strong>07</strong></td>
<td><strong>08</strong></td>
<td><strong>09</strong></td>
<td><strong>0A</strong></td>
<td><strong>0B</strong></td>
<td><strong>0C</strong></td>
<td><strong>0D</strong></td>
<td><strong>0E</strong></td>
<td><strong>0F</strong></td>
</tr>
<tr>
<td>0x00</td>
<td>FF</td>
<td>42</td>
<td>4F</td>
<td>58</td>
<td>VV</td>
<td>XX</td>
<td>RR</td>
<td>RR</td>
<td>AA</td>
<td>AA</td>
<td>AA</td>
<td>AA</td>
<td>RR</td>
<td>RR</td>
<td>RR</td>
<td>RR</td>
</tr>
<tr>
<td>0x10</td>
<td>TT</td>
<td>TT</td>
<td>TT</td>
<td>TT</td>
<td>TT</td>
<td>TT</td>
<td>TT</td>
<td>TT</td>
<td>RR</td>
<td>RR</td>
<td>RR</td>
<td>RR</td>
<td>RR</td>
<td>RR</td>
<td>RR</td>
<td>RR</td>
</tr>
</table>
<figcaption>Table 2: Header binary layout</figcaption>
</figure>
<p><strong>Legend:</strong> <span style="display: inline-block"><code>FF 42 4F 58</code></span> = magic, <span style="display: inline-block"><code>VV</code></span> = version, <span style="display: inline-block"><code>XX</code></span> = flags, <span style="display: inline-block"><code>RR</code></span> = reserved, <span style="display: inline-block"><code>AA</code></span> = alignment, <span style="display: inline-block"><code>TT</code></span> = trailer offset</p>
<h2 id="loc-22">6 Data Section</h2>
<p>The Data Section immediately follows the Header and contains the actual file content.</p>
<h3 id="loc-23">6.1 Layout</h3>
<p>File data is stored contiguously, with optional padding between files when alignment is enabled. The Data Section starts immediately after the Header. When alignment is disabled, this is offset <span style="display: inline-block"><code>0x20</code></span> (32 bytes). When alignment is enabled, padding bytes (<span style="display: inline-block"><code>0x00</code></span>) are inserted so that the first file’s data begins at an aligned offset. The Data Section extends to the trailer offset.</p>
<figure>
<table>
<tr>
<td>Offset 0x20 →</td>
<td>[Padding] <em>(if alignment > 1)</em></td>
</tr>
<tr>
<td></td>
<td><strong>File 1 Data</strong></td>
</tr>
<tr>
<td></td>
<td>[Padding] <em>(if alignment > 1)</em></td>
</tr>
<tr>
<td></td>
<td><strong>File 2 Data</strong></td>
</tr>
<tr>
<td></td>
<td>[Padding]</td>
</tr>
<tr>
<td></td>
<td>…</td>
</tr>
<tr>
<td>header.trailer →</td>
<td></td>
</tr>
</table>
<figcaption>Table 3: Data section layout</figcaption>
</figure>
<h3 id="loc-24">6.2 Alignment Padding</h3>
<p>When <span style="display: inline-block"><code>header.alignment > 1</code></span>, padding bytes (<span style="display: inline-block"><code>0x00</code></span>) are inserted before file data such that:</p>
<div>
<pre><code>file_data_offset % alignment == 0</code></pre>
</div>
<p>The amount of padding before each file is:</p>
<div>
<pre><code>padding = (alignment - (current_offset % alignment)) % alignment</code></pre>
</div>
<h2 id="loc-25">7 Trailer (Metadata)</h2>
<p>The Trailer contains all archive metadata encoded as <span style="display: inline-block"><code>BoxMetadata</code></span>.</p>
<h3 id="loc-26">7.1 BoxMetadata Structure</h3>
<div>
<pre><code>[Vec<AttrKey>: attr_keys]<br>[AttrMap: attrs]<br>[Vu64: dictionary_length]<br>[bytes: dictionary]<br>[Vec<Record>: records]</code></pre>
</div>
<p>Fields are encoded in the order shown, using the data type encodings from <span style="text-decoration: underline"><a href="#loc-6">Section 4</a></span>. The dictionary field is encoded as a Vu64 length followed by raw bytes; length=0 means no dictionary is present.</p>
<h3 id="loc-27">7.2 Attribute Key Table (attr_keys)</h3>
<p>The attribute key table provides string interning for attribute keys. Each entry consists of:</p>
<div>
<pre><code>[u8: type tag]<br>[String: key name]</code></pre>
</div>
<p>The type tag indicates the expected value type for this attribute key. See <span style="text-decoration: underline"><a href="#loc-28">Section 7.3</a></span> for type tag values.</p>
<p>Keys are referenced by their 0-based index in this table.</p>
<h3 id="loc-28">7.3 Attribute Value Types</h3>
<p><strong>Type Tag Values:</strong></p>
<table>
<thead>
<tr>
<th><strong>Value</strong></th>
<th><strong>Type</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>Bytes</td>
<td>Raw bytes, no interpretation</td>
</tr>
<tr>
<td>1</td>
<td>String</td>
<td>UTF-8 string</td>
</tr>
<tr>
<td>2</td>
<td>Json</td>
<td>UTF-8 JSON</td>
</tr>
<tr>
<td>3</td>
<td>U8</td>
<td>Single byte (u8)</td>
</tr>
<tr>
<td>4</td>
<td>Vi32</td>
<td>Zigzag-encoded i32</td>
</tr>
<tr>
<td>5</td>
<td>Vu32</td>
<td>FastVint-encoded u32</td>
</tr>
<tr>
<td>6</td>
<td>Vi64</td>
<td>Zigzag-encoded i64</td>
</tr>
<tr>
<td>7</td>
<td>Vu64</td>
<td>FastVint-encoded u64</td>
</tr>
<tr>
<td>8</td>
<td>U128</td>
<td>Fixed 16 bytes (128 bits)</td>
</tr>
<tr>
<td>9</td>
<td>U256</td>
<td>Fixed 32 bytes (256 bits)</td>
</tr>
<tr>
<td>10</td>
<td>DateTime</td>
<td>Minutes since Box epoch (2026-01-01 00:00:00 UTC), zigzag-encoded i64</td>
</tr>
<tr>
<td>11-255</td>
<td>Reserved</td>
<td>Reserved for future use</td>
</tr>
</tbody>
</table>
<p>When serializing: keys are written in symbol order (0, 1, 2, …).</p>
<p>When deserializing: strings are assigned symbols in the order read. The type tag determines how the attribute value is interpreted.</p>
<h3 id="loc-29">7.4 Archive Attributes (attrs)</h3>
<p>The <span style="display: inline-block"><code>attrs</code></span> field stores archive-level attributes as an <span style="display: inline-block"><code>AttrMap</code></span>. Common archive attributes include creation tools, comments, or custom metadata.</p>
<h3 id="loc-30">7.5 Dictionary</h3>
<p>The <span style="display: inline-block"><code>dictionary</code></span> field contains a Zstd dictionary used for compression. When present (non-empty), all records compressed with Zstd MUST use this dictionary for both compression and decompression. The dictionary is ignored for other compression methods (Stored, XZ).</p>
<p>Dictionary training is typically performed at archive creation time using <span style="display: inline-block"><code>ZDICT_trainFromBuffer()</code></span> or equivalent. A recommended dictionary size is 32KB-128KB, trained on representative samples from the archive contents totaling approximately 100× the dictionary size.</p>
<p>When <span style="display: inline-block"><code>dictionary</code></span> is empty, Zstd compression operates without a dictionary.</p>
<p>The dictionary provides two benefits:</p>
<ol>
<li value="1">Improved compression ratio, especially for small files with shared patterns</li>
<li value="2">More predictable compression ratios, which aids in estimating compressed block sizes for chunked files</li>
</ol>
<h2 id="loc-31">8 Record Types</h2>
<p>Records describe entries in the archive. Each record begins with a 1-byte type/compression identifier.</p>
<h3 id="loc-32">8.1 Type/Compression Byte</h3>
<p>The first byte of each record encodes both the record type and compression method:</p>
<div>
<pre><code>[type_compression: u8]</code></pre>
</div>
<p><strong>Layout:</strong></p>
<div>
<pre><code>Bits 0-3: Record type<br>Bits 4-7: Compression method</code></pre>
</div>
<p><strong>Record Type Values:</strong></p>
<table>
<thead>
<tr>
<th><strong>Value</strong></th>
<th><strong>Type</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td><span style="display: inline-block"><code>0x01</code></span></td>
<td>Directory</td>
<td>Directory entry</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x02</code></span></td>
<td>File</td>
<td>Regular file</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x03</code></span></td>
<td>Symlink</td>
<td>Internal symbolic link</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x0A</code></span></td>
<td>Chunked File</td>
<td>File with block-level compression</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x0B</code></span></td>
<td>External Symlink</td>
<td>Symbolic link to external target</td>
</tr>
</tbody>
</table>
<p><strong>Compression Values:</strong></p>
<table>
<thead>
<tr>
<th><strong>Value</strong></th>
<th><strong>Method</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td><span style="display: inline-block"><code>0x00</code></span></td>
<td>Stored</td>
<td>No compression</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x10</code></span></td>
<td>Zstd</td>
<td>Zstandard compression</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x20</code></span></td>
<td>XZ</td>
<td>XZ/LZMA2 compression</td>
</tr>
</tbody>
</table>
<p><strong>Combined Examples:</strong></p>
<table>
<thead>
<tr>
<th><strong>Byte</strong></th>
<th><strong>Meaning</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td><span style="display: inline-block"><code>0x01</code></span></td>
<td>Directory (compression ignored)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x02</code></span></td>
<td>File, stored (uncompressed)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x12</code></span></td>
<td>File, Zstd compressed</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x22</code></span></td>
<td>File, XZ compressed</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>0x1A</code></span></td>
<td>Chunked file, Zstd compressed blocks</td>
</tr>
</tbody>
</table>
<h3 id="loc-33">8.2 Directory Record (Type 0x01)</h3>
<p>Directories have no associated data in the Data Section.</p>
<p><strong>Structure:</strong></p>
<div>
<pre><code>[type_compression: u8] // Always 0x01<br>[String: name]<br>[AttrMap: attrs]</code></pre>
</div>
<p><strong>Binary Layout:</strong></p>
<div>
<table>
<tr>
<td><span style="display: inline-block"><code>00</code></span></td>
<td>type_compression (directory)</td>
</tr>
<tr>
<td>…</td>
<td>String: name</td>
</tr>
<tr>
<td>…</td>
<td>AttrMap: attrs</td>
</tr>
</table>
</div>
<h3 id="loc-34">8.3 File Record (Type 0x02)</h3>
<p>Files store their content in the Data Section.</p>
<p><strong>Structure:</strong></p>
<div>
<pre><code>[type_compression: u8]<br>[length: u64] // Compressed length in data section<br>[decompressed_length: u64] // Original file size<br>[data: u64] // Byte offset in archive<br>[String: name]<br>[AttrMap: attrs]</code></pre>
</div>
<p><strong>Binary Layout:</strong></p>
<div>
<table>
<tr>
<td><span style="display: inline-block"><code>TT</code></span></td>
<td>type_compression</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>LL LL LL LL LL LL LL LL</code></span></td>
<td>length (u64)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>DD DD DD DD DD DD DD DD</code></span></td>
<td>decompressed_length (u64)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>OO OO OO OO OO OO OO OO</code></span></td>
<td>data offset (u64)</td>
</tr>
<tr>
<td>…</td>
<td>String: name</td>
</tr>
<tr>
<td>…</td>
<td>AttrMap: attrs</td>
</tr>
</table>
</div>
<p>When <span style="display: inline-block"><code>compression == Stored</code></span>, <span style="display: inline-block"><code>length == decompressed_length</code></span>.</p>
<h3 id="loc-35">8.4 Symlink Record (Type 0x03)</h3>
<p>Symbolic links point to other entries within the archive.</p>
<p><strong>Structure:</strong></p>
<div>
<pre><code>[type_compression: u8] // Always 0x03<br>[String: name]<br>[Vu64: target (RecordIndex)]<br>[AttrMap: attrs]</code></pre>
</div>
<p><strong>Constraints:</strong></p>
<ul>
<li>Target MUST be a valid RecordIndex pointing to an existing record</li>
<li>Circular symlinks are not permitted</li>
<li>The target record MUST NOT be another symlink</li>
</ul>
<p><strong>Binary Layout:</strong></p>
<div>
<table>
<tr>
<td><span style="display: inline-block"><code>03</code></span></td>
<td>type_compression (symlink)</td>
</tr>
<tr>
<td>…</td>
<td>String: name</td>
</tr>
<tr>
<td>…</td>
<td>Vu64: target (RecordIndex)</td>
</tr>
<tr>
<td>…</td>
<td>AttrMap: attrs</td>
</tr>
</table>
</div>
<h3 id="loc-36">8.5 External Symlink Record (Type 0x0B)</h3>
<p>External symlinks point to paths outside the archive. This record type is only valid when the <span style="display: inline-block"><code>EXTERNAL_SYMLINKS</code></span> flag is set in the header.</p>
<p><strong>Structure:</strong></p>
<div>
<pre><code>[type_compression: u8] // Always 0x0B<br>[String: name]<br>[String: target]<br>[AttrMap: attrs]</code></pre>
</div>
<p><strong>Target Path Format:</strong></p>
<p>The target path uses BoxPath encoding (see <span style="text-decoration: underline"><a href="#loc-38">Section 9</a></span>) with one relaxation: <span style="display: inline-block"><code>..</code></span> components are permitted to allow the symlink to escape the archive root. The path is stored with <span style="display: inline-block"><code>\x1F</code></span> (Unit Separator) as the component separator.</p>
<p><strong>Constraints:</strong></p>
<ul>
<li>The header MUST have the <span style="display: inline-block"><code>EXTERNAL_SYMLINKS</code></span> flag (bit 0) set</li>
<li>Target is a relative path from the symlink’s location</li>
<li>Implementations MUST validate <span style="display: inline-block"><code>..</code></span> does not escape the extraction directory without explicit user consent</li>
</ul>
<p><strong>Binary Layout:</strong></p>
<div>
<table>
<tr>
<td><span style="display: inline-block"><code>0B</code></span></td>
<td>type_compression (external symlink)</td>
</tr>
<tr>
<td>…</td>
<td>String: name</td>
</tr>
<tr>
<td>…</td>
<td>String: target</td>
</tr>
<tr>
<td>…</td>
<td>AttrMap: attrs</td>
</tr>
</table>
</div>
<p><strong>Extraction Behavior:</strong></p>
<p>When extracting an external symlink record, implementations:</p>
<ol>
<li value="1">Convert the stored target path from BoxPath format (<span style="display: inline-block"><code>\x1F</code></span> separators) to platform path format</li>
<li value="2">Create a symbolic link at the record’s location using the converted relative target path</li>
</ol>
<p><strong>Security Considerations:</strong></p>
<p>External symlinks can point to arbitrary locations outside the extraction directory. This presents security risks:</p>
<ul>
<li>Implementations MUST require explicit user consent before extracting archives containing external symlinks.</li>
<li>This feature is opt-in at both archive creation time (setting the header flag) and extraction time.</li>
<li>Extraction tools SHOULD warn users when processing archives with external symlinks.</li>
</ul>
<h3 id="loc-37">8.6 Chunked File Record (Type 0x0A)</h3>
<p>Chunked files store content in fixed-size blocks, each independently compressed. This enables random access to large compressed files without decompressing the entire file.</p>
<p><strong>Structure:</strong></p>
<div>
<pre><code>[type_compression: u8] // 0x0A | compression_method<br>[block_size: u32] // Size of each block before compression<br>[length: u64] // Total compressed length<br>[decompressed_length: u64] // Total original file size<br>[data: u64] // Byte offset to first block<br>[String: name]<br>[AttrMap: attrs]</code></pre>
</div>
<p><strong>Binary Layout:</strong></p>
<div>
<table>
<tr>
<td><span style="display: inline-block"><code>TT</code></span></td>
<td>type_compression</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>BB BB BB BB</code></span></td>
<td>block_size (u32)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>LL LL LL LL LL LL LL LL</code></span></td>
<td>length (u64)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>DD DD DD DD DD DD DD DD</code></span></td>
<td>decompressed_length (u64)</td>
</tr>
<tr>
<td><span style="display: inline-block"><code>OO OO OO OO OO OO OO OO</code></span></td>
<td>data offset (u64)</td>
</tr>
<tr>
<td>…</td>
<td>String: name</td>
</tr>
<tr>
<td>…</td>
<td>AttrMap: attrs</td>
</tr>
</table>
</div>
<p><strong>Decompression:</strong></p>
<p>To read bytes at logical offset <span style="display: inline-block"><code>O</code></span> within a chunked file, use the Block FST lookup procedure described in <span style="text-decoration: underline"><a href="#loc-79">Section 15.5</a></span>.</p>
<p><strong>Recommended Block Size:</strong></p>
<p>A block size of 2MB (2,097,152 bytes) is RECOMMENDED. This aligns with common hugepage sizes and provides a good balance between compression ratio and random access granularity.</p>
<h2 id="loc-38">9 Path Encoding</h2>
<p>Paths in Box archives use a platform-independent encoding called BoxPath.</p>
<h3 id="loc-39">9.1 BoxPath Format</h3>
<p>BoxPath paths are sequences of components separated by <span style="display: inline-block"><code>U+001F</code></span> UNIT SEPARATOR (hex <span style="display: inline-block"><code>\x1F</code></span>).</p>
<p><strong>Properties:</strong></p>
<ul>
<li>Paths are always relative (no leading separator)</li>
<li>No trailing separator</li>
<li>Empty components are not permitted</li>
<li>Components <span style="display: inline-block"><code>.</code></span> and <span style="display: inline-block"><code>..</code></span> are not permitted</li>
<li>Components cannot contain platform path separators (<span style="display: inline-block"><code>/</code></span> or <span style="display: inline-block"><code>\</code></span>)</li>
<li>Components cannot contain null bytes</li>
</ul>
<h3 id="loc-40">9.2 Separator Choice</h3>
<p>The <span style="display: inline-block"><code>U+001F</code></span> separator was chosen because:</p>
<ol>
<li value="1">It forces explicit handling - developers cannot be lazy and treat paths as platform-native strings</li>
<li value="2">It avoids confusion with URL scheme prefixes (<span style="display: inline-block"><code>file://</code></span>, <span style="display: inline-block"><code>http://</code></span>) that would occur with <span style="display: inline-block"><code>/</code></span></li>
<li value="3">It is truly platform-agnostic - neither Unix nor Windows “owns” this separator</li>
<li value="4">It is illegal in filenames on all major platforms, preventing ambiguity</li>
</ol>
<h3 id="loc-41">9.3 Platform Conversion</h3>
<p>When extracting archives, implementations MUST convert BoxPath paths to platform-native paths:</p>
<table>
<thead>
<tr>
<th><strong>Platform</strong></th>
<th><strong>Separator</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>Unix/Linux/macOS</td>
<td><span style="display: inline-block"><code>/</code></span></td>
</tr>
<tr>
<td>Windows</td>
<td><span style="display: inline-block"><code>\</code></span></td>
</tr>
</tbody>
</table>
<p><strong>Example:</strong></p>
<div>
<pre><code>BoxPath: foo\x1Fbar\x1Fbaz.txt<br>Unix: foo/bar/baz.txt<br>Windows: foo\bar\baz.txt</code></pre>
</div>