Support gitlinks in fast-import.
[git/git.git] / Documentation / git-fast-import.txt
CommitLineData
6e411d20
SP
1git-fast-import(1)
2==================
3
4NAME
5----
7a33631f 6git-fast-import - Backend for fast Git data importers
6e411d20
SP
7
8
9SYNOPSIS
10--------
b1889c36 11frontend | 'git fast-import' [options]
6e411d20
SP
12
13DESCRIPTION
14-----------
15This program is usually not what the end user wants to run directly.
16Most end users want to use one of the existing frontend programs,
17which parses a specific type of foreign source and feeds the contents
ba020ef5 18stored there to 'git-fast-import'.
6e411d20 19
882227f1 20fast-import reads a mixed command/data stream from standard input and
6e411d20
SP
21writes one or more packfiles directly into the current repository.
22When EOF is received on standard input, fast import writes out
23updated branch and tag refs, fully updating the current repository
24with the newly imported data.
25
882227f1 26The fast-import backend itself can import into an empty repository (one that
ba020ef5 27has already been initialized by 'git-init') or incrementally
6e411d20
SP
28update an existing populated repository. Whether or not incremental
29imports are supported from a particular foreign source depends on
30the frontend program in use.
31
32
33OPTIONS
34-------
63e0c8b3
SP
35--date-format=<fmt>::
36 Specify the type of dates the frontend will supply to
882227f1 37 fast-import within `author`, `committer` and `tagger` commands.
63e0c8b3
SP
38 See ``Date Formats'' below for details about which formats
39 are supported, and their syntax.
40
7073e69e
SP
41--force::
42 Force updating modified existing branches, even if doing
43 so would cause commits to be lost (as the new commit does
44 not contain the old commit).
45
6e411d20
SP
46--max-pack-size=<n>::
47 Maximum size of each output packfile, expressed in MiB.
48 The default is 4096 (4 GiB) as that is the maximum allowed
49 packfile size (due to file format limitations). Some
50 importers may wish to lower this, such as to ensure the
51 resulting packfiles fit on CDs.
52
53--depth=<n>::
54 Maximum delta depth, for blob and tree deltification.
55 Default is 10.
56
57--active-branches=<n>::
58 Maximum number of branches to maintain active at once.
59 See ``Memory Utilization'' below for details. Default is 5.
60
61--export-marks=<file>::
62 Dumps the internal marks table to <file> when complete.
63 Marks are written one per line as `:markid SHA-1`.
64 Frontends can use this file to validate imports after they
e8438420
SP
65 have been completed, or to save the marks table across
66 incremental runs. As <file> is only opened and truncated
67 at checkpoint (or completion) the same path can also be
68 safely given to \--import-marks.
69
70--import-marks=<file>::
71 Before processing any input, load the marks specified in
72 <file>. The input file must exist, must be readable, and
73 must use the same format as produced by \--export-marks.
74 Multiple options may be supplied to import more than one
75 set of marks. If a mark is defined to different values,
76 the last file wins.
6e411d20 77
bdf1c06d
SP
78--export-pack-edges=<file>::
79 After creating a packfile, print a line of data to
80 <file> listing the filename of the packfile and the last
81 commit on each branch that was written to that packfile.
82 This information may be useful after importing projects
83 whose total object set exceeds the 4 GiB packfile limit,
84 as these commits can be used as edge points during calls
ba020ef5 85 to 'git-pack-objects'.
bdf1c06d 86
c499d768 87--quiet::
882227f1 88 Disable all non-fatal output, making fast-import silent when it
7f9d77f2 89 is successful. This option disables the output shown by
c499d768
SP
90 \--stats.
91
92--stats::
882227f1 93 Display some basic statistics about the objects fast-import has
c499d768 94 created, the packfiles they were stored into, and the
882227f1 95 memory used by fast-import during this run. Showing this output
c499d768
SP
96 is currently the default, but can be disabled with \--quiet.
97
98
6e411d20
SP
99Performance
100-----------
882227f1 101The design of fast-import allows it to import large projects in a minimum
6e411d20 102amount of memory usage and processing time. Assuming the frontend
882227f1 103is able to keep up with fast-import and feed it a constant stream of data,
6e411d20
SP
104import times for projects holding 10+ years of history and containing
105100,000+ individual commits are generally completed in just 1-2
106hours on quite modest (~$2,000 USD) hardware.
107
108Most bottlenecks appear to be in foreign source data access (the
882227f1 109source just cannot extract revisions fast enough) or disk IO (fast-import
6e411d20
SP
110writes as fast as the disk will take the data). Imports will run
111faster if the source data is stored on a different drive than the
112destination Git repository (due to less IO contention).
113
114
115Development Cost
116----------------
882227f1 117A typical frontend for fast-import tends to weigh in at approximately 200
6e411d20
SP
118lines of Perl/Python/Ruby code. Most developers have been able to
119create working importers in just a couple of hours, even though it
882227f1 120is their first exposure to fast-import, and sometimes even to Git. This is
6e411d20
SP
121an ideal situation, given that most conversion tools are throw-away
122(use once, and never look back).
123
124
125Parallel Operation
126------------------
ba020ef5 127Like 'git-push' or 'git-fetch', imports handled by fast-import are safe to
6e411d20 128run alongside parallel `git repack -a -d` or `git gc` invocations,
ba020ef5 129or any other Git operation (including 'git-prune', as loose objects
882227f1 130are never used by fast-import).
6e411d20 131
882227f1
SP
132fast-import does not lock the branch or tag refs it is actively importing.
133After the import, during its ref update phase, fast-import tests each
7073e69e
SP
134existing branch ref to verify the update will be a fast-forward
135update (the commit stored in the ref is contained in the new
136history of the commit to be written). If the update is not a
882227f1
SP
137fast-forward update, fast-import will skip updating that ref and instead
138prints a warning message. fast-import will always attempt to update all
7073e69e
SP
139branch refs, and does not stop on the first failure.
140
c499d768
SP
141Branch updates can be forced with \--force, but its recommended that
142this only be used on an otherwise quiet repository. Using \--force
7073e69e 143is not necessary for an initial import into an empty repository.
6e411d20
SP
144
145
146Technical Discussion
147--------------------
882227f1 148fast-import tracks a set of branches in memory. Any branch can be created
6e411d20
SP
149or modified at any point during the import process by sending a
150`commit` command on the input stream. This design allows a frontend
151program to process an unlimited number of branches simultaneously,
152generating commits in the order they are available from the source
153data. It also simplifies the frontend programs considerably.
154
882227f1 155fast-import does not use or alter the current working directory, or any
6e411d20
SP
156file within it. (It does however update the current Git repository,
157as referenced by `GIT_DIR`.) Therefore an import frontend may use
158the working directory for its own purposes, such as extracting file
159revisions from the foreign source. This ignorance of the working
882227f1 160directory also allows fast-import to run very quickly, as it does not
6e411d20
SP
161need to perform any costly file update operations when switching
162between branches.
163
164Input Format
165------------
166With the exception of raw file data (which Git does not interpret)
882227f1 167the fast-import input format is text (ASCII) based. This text based
6e411d20
SP
168format simplifies development and debugging of frontend programs,
169especially when a higher level language such as Perl, Python or
170Ruby is being used.
171
882227f1 172fast-import is very strict about its input. Where we say SP below we mean
6e411d20
SP
173*exactly* one space. Likewise LF means one (and only one) linefeed.
174Supplying additional whitespace characters will cause unexpected
175results, such as branch names or file names with leading or trailing
882227f1 176spaces in their name, or early termination of fast-import when it encounters
6e411d20
SP
177unexpected input.
178
401d53fa
SP
179Stream Comments
180~~~~~~~~~~~~~~~
181To aid in debugging frontends fast-import ignores any line that
182begins with `#` (ASCII pound/hash) up to and including the line
183ending `LF`. A comment line may contain any sequence of bytes
184that does not contain an LF and therefore may be used to include
185any detailed debugging information that might be specific to the
186frontend and useful when inspecting a fast-import data stream.
187
63e0c8b3
SP
188Date Formats
189~~~~~~~~~~~~
190The following date formats are supported. A frontend should select
191the format it will use for this import by passing the format name
c499d768 192in the \--date-format=<fmt> command line option.
63e0c8b3
SP
193
194`raw`::
9b92c82f 195 This is the Git native format and is `<time> SP <offutc>`.
882227f1 196 It is also fast-import's default format, if \--date-format was
63e0c8b3
SP
197 not specified.
198+
199The time of the event is specified by `<time>` as the number of
200seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
201written as an ASCII decimal integer.
202+
9b92c82f
SP
203The local offset is specified by `<offutc>` as a positive or negative
204offset from UTC. For example EST (which is 5 hours behind UTC)
205would be expressed in `<tz>` by ``-0500'' while UTC is ``+0000''.
206The local offset does not affect `<time>`; it is used only as an
207advisement to help formatting routines display the timestamp.
63e0c8b3 208+
9b92c82f
SP
209If the local offset is not available in the source material, use
210``+0000'', or the most common local offset. For example many
63e0c8b3
SP
211organizations have a CVS repository which has only ever been accessed
212by users who are located in the same location and timezone. In this
f842fdb0 213case a reasonable offset from UTC could be assumed.
63e0c8b3
SP
214+
215Unlike the `rfc2822` format, this format is very strict. Any
882227f1 216variation in formatting will cause fast-import to reject the value.
63e0c8b3
SP
217
218`rfc2822`::
219 This is the standard email format as described by RFC 2822.
220+
221An example value is ``Tue Feb 6 11:22:18 2007 -0500''. The Git
f842fdb0 222parser is accurate, but a little on the lenient side. It is the
ba020ef5 223same parser used by 'git-am' when applying patches
63e0c8b3
SP
224received from email.
225+
226Some malformed strings may be accepted as valid dates. In some of
227these cases Git will still be able to obtain the correct date from
228the malformed string. There are also some types of malformed
229strings which Git will parse wrong, and yet consider valid.
230Seriously malformed strings will be rejected.
231+
9b92c82f
SP
232Unlike the `raw` format above, the timezone/UTC offset information
233contained in an RFC 2822 date string is used to adjust the date
234value to UTC prior to storage. Therefore it is important that
235this information be as accurate as possible.
236+
f842fdb0 237If the source material uses RFC 2822 style dates,
882227f1 238the frontend should let fast-import handle the parsing and conversion
63e0c8b3
SP
239(rather than attempting to do it itself) as the Git parser has
240been well tested in the wild.
241+
242Frontends should prefer the `raw` format if the source material
f842fdb0 243already uses UNIX-epoch format, can be coaxed to give dates in that
02783075 244format, or its format is easily convertible to it, as there is no
f842fdb0 245ambiguity in parsing.
63e0c8b3
SP
246
247`now`::
248 Always use the current time and timezone. The literal
249 `now` must always be supplied for `<when>`.
250+
251This is a toy format. The current time and timezone of this system
252is always copied into the identity string at the time it is being
882227f1 253created by fast-import. There is no way to specify a different time or
63e0c8b3
SP
254timezone.
255+
256This particular format is supplied as its short to implement and
257may be useful to a process that wants to create a new commit
258right now, without needing to use a working directory or
ba020ef5 259'git-update-index'.
63e0c8b3
SP
260+
261If separate `author` and `committer` commands are used in a `commit`
262the timestamps may not match, as the system clock will be polled
263twice (once for each command). The only way to ensure that both
264author and committer identity information has the same timestamp
265is to omit `author` (thus copying from `committer`) or to use a
266date format other than `now`.
267
6e411d20
SP
268Commands
269~~~~~~~~
882227f1 270fast-import accepts several commands to update the current repository
6e411d20
SP
271and control the current import process. More detailed discussion
272(with examples) of each command follows later.
273
274`commit`::
275 Creates a new branch or updates an existing branch by
276 creating a new commit and updating the branch to point at
277 the newly created commit.
278
279`tag`::
280 Creates an annotated tag object from an existing commit or
281 branch. Lightweight tags are not supported by this command,
282 as they are not recommended for recording meaningful points
283 in time.
284
285`reset`::
286 Reset an existing branch (or a new branch) to a specific
287 revision. This command must be used to change a branch to
288 a specific revision without making a commit on it.
289
290`blob`::
291 Convert raw file data into a blob, for future use in a
292 `commit` command. This command is optional and is not
293 needed to perform an import.
294
295`checkpoint`::
882227f1 296 Forces fast-import to close the current packfile, generate its
6e411d20
SP
297 unique SHA-1 checksum and index, and start a new packfile.
298 This command is optional and is not needed to perform
299 an import.
300
ac053c02
SP
301`progress`::
302 Causes fast-import to echo the entire line to its own
303 standard output. This command is optional and is not needed
304 to perform an import.
305
6e411d20
SP
306`commit`
307~~~~~~~~
308Create or update a branch with a new commit, recording one logical
309change to the project.
310
311....
312 'commit' SP <ref> LF
313 mark?
63e0c8b3
SP
314 ('author' SP <name> SP LT <email> GT SP <when> LF)?
315 'committer' SP <name> SP LT <email> GT SP <when> LF
6e411d20
SP
316 data
317 ('from' SP <committish> LF)?
318 ('merge' SP <committish> LF)?
b6f3481b 319 (filemodify | filedelete | filecopy | filerename | filedeleteall)*
1fdb649c 320 LF?
6e411d20
SP
321....
322
323where `<ref>` is the name of the branch to make the commit on.
324Typically branch names are prefixed with `refs/heads/` in
325Git, so importing the CVS branch symbol `RELENG-1_0` would use
326`refs/heads/RELENG-1_0` for the value of `<ref>`. The value of
327`<ref>` must be a valid refname in Git. As `LF` is not valid in
328a Git refname, no quoting or escaping syntax is supported here.
329
882227f1 330A `mark` command may optionally appear, requesting fast-import to save a
6e411d20
SP
331reference to the newly created commit for future use by the frontend
332(see below for format). It is very common for frontends to mark
333every commit they create, thereby allowing future branch creation
334from any imported commit.
335
336The `data` command following `committer` must supply the commit
337message (see below for `data` command syntax). To import an empty
338commit message use a 0 length data. Commit messages are free-form
339and are not interpreted by Git. Currently they must be encoded in
882227f1 340UTF-8, as fast-import does not permit other encodings to be specified.
6e411d20 341
b6f3481b
SP
342Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`
343and `filedeleteall` commands
825769a8
SP
344may be included to update the contents of the branch prior to
345creating the commit. These commands may be supplied in any order.
02783075 346However it is recommended that a `filedeleteall` command precede
b6f3481b
SP
347all `filemodify`, `filecopy` and `filerename` commands in the same
348commit, as `filedeleteall`
825769a8 349wipes the branch clean (see below).
6e411d20 350
1fdb649c
SP
351The `LF` after the command is optional (it used to be required).
352
6e411d20
SP
353`author`
354^^^^^^^^
355An `author` command may optionally appear, if the author information
356might differ from the committer information. If `author` is omitted
882227f1 357then fast-import will automatically use the committer's information for
6e411d20
SP
358the author portion of the commit. See below for a description of
359the fields in `author`, as they are identical to `committer`.
360
361`committer`
362^^^^^^^^^^^
363The `committer` command indicates who made this commit, and when
364they made it.
365
366Here `<name>` is the person's display name (for example
367``Com M Itter'') and `<email>` is the person's email address
368(``cm@example.com''). `LT` and `GT` are the literal less-than (\x3c)
369and greater-than (\x3e) symbols. These are required to delimit
370the email address from the other fields in the line. Note that
371`<name>` is free-form and may contain any sequence of bytes, except
372`LT` and `LF`. It is typically UTF-8 encoded.
373
63e0c8b3 374The time of the change is specified by `<when>` using the date format
c499d768 375that was selected by the \--date-format=<fmt> command line option.
63e0c8b3
SP
376See ``Date Formats'' above for the set of supported formats, and
377their syntax.
6e411d20
SP
378
379`from`
380^^^^^^
ea5e370a
SP
381The `from` command is used to specify the commit to initialize
382this branch from. This revision will be the first ancestor of the
383new commit.
384
385Omitting the `from` command in the first commit of a new branch
386will cause fast-import to create that commit with no ancestor. This
387tends to be desired only for the initial commit of a project.
9b33fa08
EB
388If the frontend creates all files from scratch when making a new
389branch, a `merge` command may be used instead of `from` to start
390the commit with an empty tree.
ea5e370a
SP
391Omitting the `from` command on existing branches is usually desired,
392as the current commit on that branch is automatically assumed to
393be the first ancestor of the new commit.
6e411d20
SP
394
395As `LF` is not valid in a Git refname or SHA-1 expression, no
396quoting or escaping syntax is supported within `<committish>`.
397
398Here `<committish>` is any of the following:
399
882227f1
SP
400* The name of an existing branch already in fast-import's internal branch
401 table. If fast-import doesn't know the name, its treated as a SHA-1
6e411d20
SP
402 expression.
403
404* A mark reference, `:<idnum>`, where `<idnum>` is the mark number.
405+
882227f1 406The reason fast-import uses `:` to denote a mark reference is this character
6e411d20 407is not legal in a Git branch name. The leading `:` makes it easy
02783075 408to distinguish between the mark 42 (`:42`) and the branch 42 (`42`
6e411d20
SP
409or `refs/heads/42`), or an abbreviated SHA-1 which happened to
410consist only of base-10 digits.
411+
412Marks must be declared (via `mark`) before they can be used.
413
414* A complete 40 byte or abbreviated commit SHA-1 in hex.
415
416* Any valid Git SHA-1 expression that resolves to a commit. See
5162e697 417 ``SPECIFYING REVISIONS'' in linkgit:git-rev-parse[1] for details.
6e411d20
SP
418
419The special case of restarting an incremental import from the
420current branch value should be written as:
421----
422 from refs/heads/branch^0
423----
882227f1 424The `{caret}0` suffix is necessary as fast-import does not permit a branch to
6e411d20 425start from itself, and the branch is created in memory before the
209f1298 426`from` command is even read from the input. Adding `{caret}0` will force
882227f1 427fast-import to resolve the commit through Git's revision parsing library,
6e411d20
SP
428rather than its internal branch table, thereby loading in the
429existing value of the branch.
430
431`merge`
432^^^^^^^
9b33fa08
EB
433Includes one additional ancestor commit. If the `from` command is
434omitted when creating a new branch, the first `merge` commit will be
435the first ancestor of the current commit, and the branch will start
436out with no files. An unlimited number of `merge` commands per
882227f1 437commit are permitted by fast-import, thereby establishing an n-way merge.
6e411d20
SP
438However Git's other tools never create commits with more than 15
439additional ancestors (forming a 16-way merge). For this reason
440it is suggested that frontends do not use more than 15 `merge`
9b33fa08 441commands per commit; 16, if starting a new, empty branch.
6e411d20
SP
442
443Here `<committish>` is any of the commit specification expressions
444also accepted by `from` (see above).
445
446`filemodify`
ef94edb5 447^^^^^^^^^^^^
6e411d20
SP
448Included in a `commit` command to add a new file or change the
449content of an existing file. This command has two different means
450of specifying the content of the file.
451
452External data format::
453 The data content for the file was already supplied by a prior
454 `blob` command. The frontend just needs to connect it.
455+
456....
457 'M' SP <mode> SP <dataref> SP <path> LF
458....
459+
460Here `<dataref>` can be either a mark reference (`:<idnum>`)
461set by a prior `blob` command, or a full 40-byte SHA-1 of an
462existing Git blob object.
463
464Inline data format::
465 The data content for the file has not been supplied yet.
466 The frontend wants to supply it as part of this modify
467 command.
468+
469....
470 'M' SP <mode> SP 'inline' SP <path> LF
471 data
472....
473+
474See below for a detailed description of the `data` command.
475
476In both formats `<mode>` is the type of file entry, specified
477in octal. Git only supports the following modes:
478
479* `100644` or `644`: A normal (not-executable) file. The majority
480 of files in most projects use this mode. If in doubt, this is
481 what you want.
482* `100755` or `755`: A normal, but executable, file.
9981b6d9 483* `120000`: A symlink, the content of the file will be the link target.
03db4525
AG
484* `160000`: A gitlink, SHA-1 of the object refers to a commit in
485 another repository. Git links can only be specified by SHA or through
486 a commit mark. They are used to implement submodules.
6e411d20
SP
487
488In both formats `<path>` is the complete path of the file to be added
489(if not already existing) or modified (if already existing).
490
c4431d38 491A `<path>` string must use UNIX-style directory separators (forward
6e411d20
SP
492slash `/`), may contain any byte other than `LF`, and must not
493start with double quote (`"`).
494
495If an `LF` or double quote must be encoded into `<path>` shell-style
496quoting should be used, e.g. `"path/with\n and \" in it"`.
497
02783075 498The value of `<path>` must be in canonical form. That is it must not:
6e411d20
SP
499
500* contain an empty directory component (e.g. `foo//bar` is invalid),
c4431d38
JK
501* end with a directory separator (e.g. `foo/` is invalid),
502* start with a directory separator (e.g. `/foo` is invalid),
6e411d20
SP
503* contain the special component `.` or `..` (e.g. `foo/./bar` and
504 `foo/../bar` are invalid).
505
506It is recommended that `<path>` always be encoded using UTF-8.
507
6e411d20 508`filedelete`
ef94edb5 509^^^^^^^^^^^^
512e44b2
SP
510Included in a `commit` command to remove a file or recursively
511delete an entire directory from the branch. If the file or directory
512removal makes its parent directory empty, the parent directory will
6e411d20
SP
513be automatically removed too. This cascades up the tree until the
514first non-empty directory or the root is reached.
515
516....
517 'D' SP <path> LF
518....
519
512e44b2
SP
520here `<path>` is the complete path of the file or subdirectory to
521be removed from the branch.
6e411d20
SP
522See `filemodify` above for a detailed description of `<path>`.
523
b6f3481b
SP
524`filecopy`
525^^^^^^^^^^^^
526Recursively copies an existing file or subdirectory to a different
527location within the branch. The existing file or directory must
528exist. If the destination exists it will be completely replaced
529by the content copied from the source.
530
531....
532 'C' SP <path> SP <path> LF
533....
534
535here the first `<path>` is the source location and the second
536`<path>` is the destination. See `filemodify` above for a detailed
537description of what `<path>` may look like. To use a source path
538that contains SP the path must be quoted.
539
540A `filecopy` command takes effect immediately. Once the source
541location has been copied to the destination any future commands
542applied to the source location will not impact the destination of
543the copy.
544
f39a946a
SP
545`filerename`
546^^^^^^^^^^^^
547Renames an existing file or subdirectory to a different location
548within the branch. The existing file or directory must exist. If
549the destination exists it will be replaced by the source directory.
550
551....
552 'R' SP <path> SP <path> LF
553....
554
555here the first `<path>` is the source location and the second
556`<path>` is the destination. See `filemodify` above for a detailed
557description of what `<path>` may look like. To use a source path
558that contains SP the path must be quoted.
559
560A `filerename` command takes effect immediately. Once the source
561location has been renamed to the destination any future commands
562applied to the source location will create new files there and not
563impact the destination of the rename.
564
b6f3481b
SP
565Note that a `filerename` is the same as a `filecopy` followed by a
566`filedelete` of the source location. There is a slight performance
567advantage to using `filerename`, but the advantage is so small
568that it is never worth trying to convert a delete/add pair in
569source material into a rename for fast-import. This `filerename`
570command is provided just to simplify frontends that already have
571rename information and don't want bother with decomposing it into a
572`filecopy` followed by a `filedelete`.
573
825769a8
SP
574`filedeleteall`
575^^^^^^^^^^^^^^^
576Included in a `commit` command to remove all files (and also all
577directories) from the branch. This command resets the internal
578branch structure to have no files in it, allowing the frontend
579to subsequently add all interesting files from scratch.
580
581....
582 'deleteall' LF
583....
584
585This command is extremely useful if the frontend does not know
586(or does not care to know) what files are currently on the branch,
587and therefore cannot generate the proper `filedelete` commands to
588update the content.
589
590Issuing a `filedeleteall` followed by the needed `filemodify`
591commands to set the correct content will produce the same results
592as sending only the needed `filemodify` and `filedelete` commands.
882227f1 593The `filedeleteall` approach may however require fast-import to use slightly
825769a8
SP
594more memory per active branch (less than 1 MiB for even most large
595projects); so frontends that can easily obtain only the affected
596paths for a commit are encouraged to do so.
597
6e411d20
SP
598`mark`
599~~~~~~
882227f1 600Arranges for fast-import to save a reference to the current object, allowing
6e411d20
SP
601the frontend to recall this object at a future point in time, without
602knowing its SHA-1. Here the current object is the object creation
603command the `mark` command appears within. This can be `commit`,
604`tag`, and `blob`, but `commit` is the most common usage.
605
606....
607 'mark' SP ':' <idnum> LF
608....
609
610where `<idnum>` is the number assigned by the frontend to this mark.
ef94edb5
SP
611The value of `<idnum>` is expressed as an ASCII decimal integer.
612The value 0 is reserved and cannot be used as
6e411d20
SP
613a mark. Only values greater than or equal to 1 may be used as marks.
614
615New marks are created automatically. Existing marks can be moved
616to another object simply by reusing the same `<idnum>` in another
617`mark` command.
618
619`tag`
620~~~~~
621Creates an annotated tag referring to a specific commit. To create
622lightweight (non-annotated) tags see the `reset` command below.
623
624....
625 'tag' SP <name> LF
626 'from' SP <committish> LF
63e0c8b3 627 'tagger' SP <name> SP LT <email> GT SP <when> LF
6e411d20 628 data
6e411d20
SP
629....
630
631where `<name>` is the name of the tag to create.
632
633Tag names are automatically prefixed with `refs/tags/` when stored
634in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would
882227f1 635use just `RELENG-1_0-FINAL` for `<name>`, and fast-import will write the
6e411d20
SP
636corresponding ref as `refs/tags/RELENG-1_0-FINAL`.
637
638The value of `<name>` must be a valid refname in Git and therefore
639may contain forward slashes. As `LF` is not valid in a Git refname,
640no quoting or escaping syntax is supported here.
641
642The `from` command is the same as in the `commit` command; see
643above for details.
644
645The `tagger` command uses the same format as `committer` within
646`commit`; again see above for details.
647
648The `data` command following `tagger` must supply the annotated tag
649message (see below for `data` command syntax). To import an empty
650tag message use a 0 length data. Tag messages are free-form and are
651not interpreted by Git. Currently they must be encoded in UTF-8,
882227f1 652as fast-import does not permit other encodings to be specified.
6e411d20 653
882227f1 654Signing annotated tags during import from within fast-import is not
6e411d20
SP
655supported. Trying to include your own PGP/GPG signature is not
656recommended, as the frontend does not (easily) have access to the
657complete set of bytes which normally goes into such a signature.
882227f1 658If signing is required, create lightweight tags from within fast-import with
6e411d20 659`reset`, then create the annotated versions of those tags offline
ba020ef5 660with the standard 'git-tag' process.
6e411d20
SP
661
662`reset`
663~~~~~~~
664Creates (or recreates) the named branch, optionally starting from
665a specific revision. The reset command allows a frontend to issue
666a new `from` command for an existing branch, or to create a new
667branch from an existing commit without creating a new commit.
668
669....
670 'reset' SP <ref> LF
671 ('from' SP <committish> LF)?
1fdb649c 672 LF?
6e411d20
SP
673....
674
675For a detailed description of `<ref>` and `<committish>` see above
676under `commit` and `from`.
677
1fdb649c
SP
678The `LF` after the command is optional (it used to be required).
679
6e411d20
SP
680The `reset` command can also be used to create lightweight
681(non-annotated) tags. For example:
682
683====
684 reset refs/tags/938
685 from :938
686====
687
688would create the lightweight tag `refs/tags/938` referring to
689whatever commit mark `:938` references.
690
691`blob`
692~~~~~~
693Requests writing one file revision to the packfile. The revision
694is not connected to any commit; this connection must be formed in
695a subsequent `commit` command by referencing the blob through an
696assigned mark.
697
698....
699 'blob' LF
700 mark?
701 data
702....
703
704The mark command is optional here as some frontends have chosen
705to generate the Git SHA-1 for the blob on their own, and feed that
706directly to `commit`. This is typically more work than its worth
707however, as marks are inexpensive to store and easy to use.
708
709`data`
710~~~~~~
711Supplies raw data (for use as blob/file content, commit messages, or
882227f1 712annotated tag messages) to fast-import. Data can be supplied using an exact
6e411d20
SP
713byte count or delimited with a terminating line. Real frontends
714intended for production-quality conversions should always use the
715exact byte count format, as it is more robust and performs better.
882227f1 716The delimited format is intended primarily for testing fast-import.
6e411d20 717
401d53fa
SP
718Comment lines appearing within the `<raw>` part of `data` commands
719are always taken to be part of the body of the data and are therefore
720never ignored by fast-import. This makes it safe to import any
721file/message content whose lines might start with `#`.
722
ef94edb5
SP
723Exact byte count format::
724 The frontend must specify the number of bytes of data.
725+
6e411d20
SP
726....
727 'data' SP <count> LF
2c570cde 728 <raw> LF?
6e411d20 729....
ef94edb5 730+
6e411d20 731where `<count>` is the exact number of bytes appearing within
ef94edb5
SP
732`<raw>`. The value of `<count>` is expressed as an ASCII decimal
733integer. The `LF` on either side of `<raw>` is not
6e411d20 734included in `<count>` and will not be included in the imported data.
2c570cde
SP
735+
736The `LF` after `<raw>` is optional (it used to be required) but
737recommended. Always including it makes debugging a fast-import
738stream easier as the next command always starts in column 0
739of the next line, even if `<raw>` did not end with an `LF`.
6e411d20 740
ef94edb5
SP
741Delimited format::
742 A delimiter string is used to mark the end of the data.
882227f1 743 fast-import will compute the length by searching for the delimiter.
02783075 744 This format is primarily useful for testing and is not
ef94edb5
SP
745 recommended for real data.
746+
6e411d20
SP
747....
748 'data' SP '<<' <delim> LF
749 <raw> LF
750 <delim> LF
2c570cde 751 LF?
6e411d20 752....
ef94edb5 753+
6e411d20
SP
754where `<delim>` is the chosen delimiter string. The string `<delim>`
755must not appear on a line by itself within `<raw>`, as otherwise
882227f1 756fast-import will think the data ends earlier than it really does. The `LF`
6e411d20
SP
757immediately trailing `<raw>` is part of `<raw>`. This is one of
758the limitations of the delimited format, it is impossible to supply
759a data chunk which does not have an LF as its last byte.
2c570cde
SP
760+
761The `LF` after `<delim> LF` is optional (it used to be required).
6e411d20
SP
762
763`checkpoint`
764~~~~~~~~~~~~
882227f1 765Forces fast-import to close the current packfile, start a new one, and to
820b9310 766save out all current branch refs, tags and marks.
6e411d20
SP
767
768....
769 'checkpoint' LF
1fdb649c 770 LF?
6e411d20
SP
771....
772
882227f1 773Note that fast-import automatically switches packfiles when the current
820b9310 774packfile reaches \--max-pack-size, or 4 GiB, whichever limit is
882227f1 775smaller. During an automatic packfile switch fast-import does not update
820b9310
SP
776the branch refs, tags or marks.
777
778As a `checkpoint` can require a significant amount of CPU time and
779disk IO (to compute the overall pack SHA-1 checksum, generate the
780corresponding index file, and update the refs) it can easily take
781several minutes for a single `checkpoint` command to complete.
782
783Frontends may choose to issue checkpoints during extremely large
784and long running imports, or when they need to allow another Git
785process access to a branch. However given that a 30 GiB Subversion
882227f1 786repository can be loaded into Git through fast-import in about 3 hours,
820b9310
SP
787explicit checkpointing may not be necessary.
788
1fdb649c 789The `LF` after the command is optional (it used to be required).
820b9310 790
ac053c02
SP
791`progress`
792~~~~~~~~~~
793Causes fast-import to print the entire `progress` line unmodified to
794its standard output channel (file descriptor 1) when the command is
795processed from the input stream. The command otherwise has no impact
796on the current import, or on any of fast-import's internal state.
797
798....
799 'progress' SP <any> LF
800 LF?
801....
802
803The `<any>` part of the command may contain any sequence of bytes
804that does not contain `LF`. The `LF` after the command is optional.
805Callers may wish to process the output through a tool such as sed to
806remove the leading part of the line, for example:
807
808====
b1889c36 809 frontend | git fast-import | sed 's/^progress //'
ac053c02
SP
810====
811
812Placing a `progress` command immediately after a `checkpoint` will
813inform the reader when the `checkpoint` has been completed and it
814can safely access the refs that fast-import updated.
815
e7e5170f
SP
816Crash Reports
817-------------
818If fast-import is supplied invalid input it will terminate with a
819non-zero exit status and create a crash report in the top level of
820the Git repository it was importing into. Crash reports contain
821a snapshot of the internal fast-import state as well as the most
822recent commands that lead up to the crash.
823
824All recent commands (including stream comments, file changes and
825progress commands) are shown in the command history within the crash
826report, but raw file data and commit messages are excluded from the
827crash report. This exclusion saves space within the report file
828and reduces the amount of buffering that fast-import must perform
829during execution.
830
831After writing a crash report fast-import will close the current
832packfile and export the marks table. This allows the frontend
833developer to inspect the repository state and resume the import from
834the point where it crashed. The modified branches and tags are not
835updated during a crash, as the import did not complete successfully.
836Branch and tag information can be found in the crash report and
837must be applied manually if the update is needed.
838
839An example crash:
840
841====
842 $ cat >in <<END_OF_INPUT
843 # my very first test commit
844 commit refs/heads/master
845 committer Shawn O. Pearce <spearce> 19283 -0400
846 # who is that guy anyway?
847 data <<EOF
848 this is my commit
849 EOF
850 M 644 inline .gitignore
851 data <<EOF
852 .gitignore
853 EOF
854 M 777 inline bob
855 END_OF_INPUT
856
b1889c36 857 $ git fast-import <in
e7e5170f
SP
858 fatal: Corrupt mode: M 777 inline bob
859 fast-import: dumping crash report to .git/fast_import_crash_8434
860
861 $ cat .git/fast_import_crash_8434
862 fast-import crash report:
863 fast-import process: 8434
864 parent process : 1391
865 at Sat Sep 1 00:58:12 2007
866
867 fatal: Corrupt mode: M 777 inline bob
868
869 Most Recent Commands Before Crash
870 ---------------------------------
871 # my very first test commit
872 commit refs/heads/master
873 committer Shawn O. Pearce <spearce> 19283 -0400
874 # who is that guy anyway?
875 data <<EOF
876 M 644 inline .gitignore
877 data <<EOF
878 * M 777 inline bob
879
880 Active Branch LRU
881 -----------------
882 active_branches = 1 cur, 5 max
883
884 pos clock name
885 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
886 1) 0 refs/heads/master
887
888 Inactive Branches
889 -----------------
890 refs/heads/master:
891 status : active loaded dirty
892 tip commit : 0000000000000000000000000000000000000000
893 old tree : 0000000000000000000000000000000000000000
894 cur tree : 0000000000000000000000000000000000000000
895 commit clock: 0
896 last pack :
897
898
899 -------------------
900 END OF CRASH REPORT
901====
902
bdd9f424
SP
903Tips and Tricks
904---------------
905The following tips and tricks have been collected from various
882227f1 906users of fast-import, and are offered here as suggestions.
bdd9f424
SP
907
908Use One Mark Per Commit
909~~~~~~~~~~~~~~~~~~~~~~~
910When doing a repository conversion, use a unique mark per commit
911(`mark :<n>`) and supply the \--export-marks option on the command
882227f1 912line. fast-import will dump a file which lists every mark and the Git
bdd9f424
SP
913object SHA-1 that corresponds to it. If the frontend can tie
914the marks back to the source repository, it is easy to verify the
915accuracy and completeness of the import by comparing each Git
916commit to the corresponding source revision.
917
918Coming from a system such as Perforce or Subversion this should be
882227f1 919quite simple, as the fast-import mark can also be the Perforce changeset
bdd9f424
SP
920number or the Subversion revision number.
921
922Freely Skip Around Branches
923~~~~~~~~~~~~~~~~~~~~~~~~~~~
924Don't bother trying to optimize the frontend to stick to one branch
925at a time during an import. Although doing so might be slightly
882227f1 926faster for fast-import, it tends to increase the complexity of the frontend
bdd9f424
SP
927code considerably.
928
882227f1 929The branch LRU builtin to fast-import tends to behave very well, and the
bdd9f424
SP
930cost of activating an inactive branch is so low that bouncing around
931between branches has virtually no impact on import performance.
932
c7346156
SP
933Handling Renames
934~~~~~~~~~~~~~~~~
935When importing a renamed file or directory, simply delete the old
936name(s) and modify the new name(s) during the corresponding commit.
937Git performs rename detection after-the-fact, rather than explicitly
938during a commit.
939
bdd9f424
SP
940Use Tag Fixup Branches
941~~~~~~~~~~~~~~~~~~~~~~
942Some other SCM systems let the user create a tag from multiple
943files which are not from the same commit/changeset. Or to create
944tags which are a subset of the files available in the repository.
945
946Importing these tags as-is in Git is impossible without making at
947least one commit which ``fixes up'' the files to match the content
882227f1 948of the tag. Use fast-import's `reset` command to reset a dummy branch
bdd9f424
SP
949outside of your normal branch space to the base commit for the tag,
950then commit one or more file fixup commits, and finally tag the
951dummy branch.
952
953For example since all normal branches are stored under `refs/heads/`
954name the tag fixup branch `TAG_FIXUP`. This way it is impossible for
955the fixup branch used by the importer to have namespace conflicts
956with real branches imported from the source (the name `TAG_FIXUP`
957is not `refs/heads/TAG_FIXUP`).
958
959When committing fixups, consider using `merge` to connect the
960commit(s) which are supplying file revisions to the fixup branch.
ba020ef5 961Doing so will allow tools such as 'git-blame' to track
bdd9f424
SP
962through the real commit history and properly annotate the source
963files.
964
882227f1 965After fast-import terminates the frontend will need to do `rm .git/TAG_FIXUP`
bdd9f424
SP
966to remove the dummy branch.
967
968Import Now, Repack Later
969~~~~~~~~~~~~~~~~~~~~~~~~
882227f1 970As soon as fast-import completes the Git repository is completely valid
02783075 971and ready for use. Typically this takes only a very short time,
bdd9f424
SP
972even for considerably large projects (100,000+ commits).
973
974However repacking the repository is necessary to improve data
975locality and access performance. It can also take hours on extremely
976large projects (especially if -f and a large \--window parameter is
977used). Since repacking is safe to run alongside readers and writers,
978run the repack in the background and let it finish when it finishes.
979There is no reason to wait to explore your new Git project!
980
981If you choose to wait for the repack, don't try to run benchmarks
882227f1 982or performance tests until repacking is completed. fast-import outputs
bdd9f424
SP
983suboptimal packfiles that are simply never seen in real use
984situations.
985
986Repacking Historical Data
987~~~~~~~~~~~~~~~~~~~~~~~~~
988If you are repacking very old imported data (e.g. older than the
989last year), consider expending some extra CPU time and supplying
ba020ef5 990\--window=50 (or higher) when you run 'git-repack'.
bdd9f424
SP
991This will take longer, but will also produce a smaller packfile.
992You only need to expend the effort once, and everyone using your
993project will benefit from the smaller repository.
994
ac053c02
SP
995Include Some Progress Messages
996~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
997Every once in a while have your frontend emit a `progress` message
998to fast-import. The contents of the messages are entirely free-form,
999so one suggestion would be to output the current month and year
1000each time the current commit date moves into the next month.
1001Your users will feel better knowing how much of the data stream
1002has been processed.
1003
bdd9f424 1004
6e411d20
SP
1005Packfile Optimization
1006---------------------
882227f1 1007When packing a blob fast-import always attempts to deltify against the last
6e411d20
SP
1008blob written. Unless specifically arranged for by the frontend,
1009this will probably not be a prior version of the same file, so the
1010generated delta will not be the smallest possible. The resulting
1011packfile will be compressed, but will not be optimal.
1012
1013Frontends which have efficient access to all revisions of a
1014single file (for example reading an RCS/CVS ,v file) can choose
1015to supply all revisions of that file as a sequence of consecutive
882227f1 1016`blob` commands. This allows fast-import to deltify the different file
6e411d20
SP
1017revisions against each other, saving space in the final packfile.
1018Marks can be used to later identify individual file revisions during
1019a sequence of `commit` commands.
1020
882227f1
SP
1021The packfile(s) created by fast-import do not encourage good disk access
1022patterns. This is caused by fast-import writing the data in the order
6e411d20
SP
1023it is received on standard input, while Git typically organizes
1024data within packfiles to make the most recent (current tip) data
1025appear before historical data. Git also clusters commits together,
1026speeding up revision traversal through better cache locality.
1027
1028For this reason it is strongly recommended that users repack the
882227f1 1029repository with `git repack -a -d` after fast-import completes, allowing
6e411d20
SP
1030Git to reorganize the packfiles for faster data access. If blob
1031deltas are suboptimal (see above) then also adding the `-f` option
1032to force recomputation of all deltas can significantly reduce the
1033final packfile size (30-50% smaller can be quite typical).
1034
bdd9f424 1035
6e411d20
SP
1036Memory Utilization
1037------------------
882227f1 1038There are a number of factors which affect how much memory fast-import
6e411d20 1039requires to perform an import. Like critical sections of core
02783075
BH
1040Git, fast-import uses its own memory allocators to amortize any overheads
1041associated with malloc. In practice fast-import tends to amortize any
6e411d20
SP
1042malloc overheads to 0, due to its use of large block allocations.
1043
1044per object
1045~~~~~~~~~~
882227f1 1046fast-import maintains an in-memory structure for every object written in
6e411d20
SP
1047this execution. On a 32 bit system the structure is 32 bytes,
1048on a 64 bit system the structure is 40 bytes (due to the larger
1049pointer sizes). Objects in the table are not deallocated until
882227f1 1050fast-import terminates. Importing 2 million objects on a 32 bit system
6e411d20
SP
1051will require approximately 64 MiB of memory.
1052
1053The object table is actually a hashtable keyed on the object name
882227f1 1054(the unique SHA-1). This storage configuration allows fast-import to reuse
6e411d20
SP
1055an existing or already written object and avoid writing duplicates
1056to the output packfile. Duplicate blobs are surprisingly common
1057in an import, typically due to branch merges in the source.
1058
1059per mark
1060~~~~~~~~
1061Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
1062bytes, depending on pointer size) per mark. Although the array
1063is sparse, frontends are still strongly encouraged to use marks
1064between 1 and n, where n is the total number of marks required for
1065this import.
1066
1067per branch
1068~~~~~~~~~~
1069Branches are classified as active and inactive. The memory usage
1070of the two classes is significantly different.
1071
1072Inactive branches are stored in a structure which uses 96 or 120
1073bytes (32 bit or 64 bit systems, respectively), plus the length of
882227f1 1074the branch name (typically under 200 bytes), per branch. fast-import will
6e411d20
SP
1075easily handle as many as 10,000 inactive branches in under 2 MiB
1076of memory.
1077
1078Active branches have the same overhead as inactive branches, but
1079also contain copies of every tree that has been recently modified on
1080that branch. If subtree `include` has not been modified since the
1081branch became active, its contents will not be loaded into memory,
1082but if subtree `src` has been modified by a commit since the branch
1083became active, then its contents will be loaded in memory.
1084
1085As active branches store metadata about the files contained on that
1086branch, their in-memory storage size can grow to a considerable size
1087(see below).
1088
882227f1 1089fast-import automatically moves active branches to inactive status based on
6e411d20
SP
1090a simple least-recently-used algorithm. The LRU chain is updated on
1091each `commit` command. The maximum number of active branches can be
c499d768 1092increased or decreased on the command line with \--active-branches=.
6e411d20
SP
1093
1094per active tree
1095~~~~~~~~~~~~~~~
1096Trees (aka directories) use just 12 bytes of memory on top of the
1097memory required for their entries (see ``per active file'' below).
02783075 1098The cost of a tree is virtually 0, as its overhead amortizes out
6e411d20
SP
1099over the individual file entries.
1100
1101per active file entry
1102~~~~~~~~~~~~~~~~~~~~~
1103Files (and pointers to subtrees) within active trees require 52 or 64
1104bytes (32/64 bit platforms) per entry. To conserve space, file and
1105tree names are pooled in a common string table, allowing the filename
1106``Makefile'' to use just 16 bytes (after including the string header
1107overhead) no matter how many times it occurs within the project.
1108
1109The active branch LRU, when coupled with the filename string pool
882227f1 1110and lazy loading of subtrees, allows fast-import to efficiently import
6e411d20
SP
1111projects with 2,000+ branches and 45,114+ files in a very limited
1112memory footprint (less than 2.7 MiB per active branch).
1113
1114
1115Author
1116------
1117Written by Shawn O. Pearce <spearce@spearce.org>.
1118
1119Documentation
1120--------------
1121Documentation by Shawn O. Pearce <spearce@spearce.org>.
1122
1123GIT
1124---
9e1f0a85 1125Part of the linkgit:git[1] suite