Merge branch 'jt/connectivity-check-after-unshallow'
[git/git.git] / Documentation / technical / protocol-v2.txt
1 Git Wire Protocol, Version 2
2 ==============================
3
4 This document presents a specification for a version 2 of Git's wire
5 protocol. Protocol v2 will improve upon v1 in the following ways:
6
7 * Instead of multiple service names, multiple commands will be
8 supported by a single service
9 * Easily extendable as capabilities are moved into their own section
10 of the protocol, no longer being hidden behind a NUL byte and
11 limited by the size of a pkt-line
12 * Separate out other information hidden behind NUL bytes (e.g. agent
13 string as a capability and symrefs can be requested using 'ls-refs')
14 * Reference advertisement will be omitted unless explicitly requested
15 * ls-refs command to explicitly request some refs
16 * Designed with http and stateless-rpc in mind. With clear flush
17 semantics the http remote helper can simply act as a proxy
18
19 In protocol v2 communication is command oriented. When first contacting a
20 server a list of capabilities will advertised. Some of these capabilities
21 will be commands which a client can request be executed. Once a command
22 has completed, a client can reuse the connection and request that other
23 commands be executed.
24
25 Packet-Line Framing
26 ---------------------
27
28 All communication is done using packet-line framing, just as in v1. See
29 `Documentation/technical/pack-protocol.txt` and
30 `Documentation/technical/protocol-common.txt` for more information.
31
32 In protocol v2 these special packets will have the following semantics:
33
34 * '0000' Flush Packet (flush-pkt) - indicates the end of a message
35 * '0001' Delimiter Packet (delim-pkt) - separates sections of a message
36
37 Initial Client Request
38 ------------------------
39
40 In general a client can request to speak protocol v2 by sending
41 `version=2` through the respective side-channel for the transport being
42 used which inevitably sets `GIT_PROTOCOL`. More information can be
43 found in `pack-protocol.txt` and `http-protocol.txt`. In all cases the
44 response from the server is the capability advertisement.
45
46 Git Transport
47 ~~~~~~~~~~~~~~~
48
49 When using the git:// transport, you can request to use protocol v2 by
50 sending "version=2" as an extra parameter:
51
52 003egit-upload-pack /project.git\0host=myserver.com\0\0version=2\0
53
54 SSH and File Transport
55 ~~~~~~~~~~~~~~~~~~~~~~~~
56
57 When using either the ssh:// or file:// transport, the GIT_PROTOCOL
58 environment variable must be set explicitly to include "version=2".
59
60 HTTP Transport
61 ~~~~~~~~~~~~~~~~
62
63 When using the http:// or https:// transport a client makes a "smart"
64 info/refs request as described in `http-protocol.txt` and requests that
65 v2 be used by supplying "version=2" in the `Git-Protocol` header.
66
67 C: GET $GIT_URL/info/refs?service=git-upload-pack HTTP/1.0
68 C: Git-Protocol: version=2
69
70 A v2 server would reply:
71
72 S: 200 OK
73 S: <Some headers>
74 S: ...
75 S:
76 S: 000eversion 2\n
77 S: <capability-advertisement>
78
79 Subsequent requests are then made directly to the service
80 `$GIT_URL/git-upload-pack`. (This works the same for git-receive-pack).
81
82 Capability Advertisement
83 --------------------------
84
85 A server which decides to communicate (based on a request from a client)
86 using protocol version 2, notifies the client by sending a version string
87 in its initial response followed by an advertisement of its capabilities.
88 Each capability is a key with an optional value. Clients must ignore all
89 unknown keys. Semantics of unknown values are left to the definition of
90 each key. Some capabilities will describe commands which can be requested
91 to be executed by the client.
92
93 capability-advertisement = protocol-version
94 capability-list
95 flush-pkt
96
97 protocol-version = PKT-LINE("version 2" LF)
98 capability-list = *capability
99 capability = PKT-LINE(key[=value] LF)
100
101 key = 1*(ALPHA | DIGIT | "-_")
102 value = 1*(ALPHA | DIGIT | " -_.,?\/{}[]()<>!@#$%^&*+=:;")
103
104 Command Request
105 -----------------
106
107 After receiving the capability advertisement, a client can then issue a
108 request to select the command it wants with any particular capabilities
109 or arguments. There is then an optional section where the client can
110 provide any command specific parameters or queries. Only a single
111 command can be requested at a time.
112
113 request = empty-request | command-request
114 empty-request = flush-pkt
115 command-request = command
116 capability-list
117 [command-args]
118 flush-pkt
119 command = PKT-LINE("command=" key LF)
120 command-args = delim-pkt
121 *command-specific-arg
122
123 command-specific-args are packet line framed arguments defined by
124 each individual command.
125
126 The server will then check to ensure that the client's request is
127 comprised of a valid command as well as valid capabilities which were
128 advertised. If the request is valid the server will then execute the
129 command. A server MUST wait till it has received the client's entire
130 request before issuing a response. The format of the response is
131 determined by the command being executed, but in all cases a flush-pkt
132 indicates the end of the response.
133
134 When a command has finished, and the client has received the entire
135 response from the server, a client can either request that another
136 command be executed or can terminate the connection. A client may
137 optionally send an empty request consisting of just a flush-pkt to
138 indicate that no more requests will be made.
139
140 Capabilities
141 --------------
142
143 There are two different types of capabilities: normal capabilities,
144 which can be used to to convey information or alter the behavior of a
145 request, and commands, which are the core actions that a client wants to
146 perform (fetch, push, etc).
147
148 Protocol version 2 is stateless by default. This means that all commands
149 must only last a single round and be stateless from the perspective of the
150 server side, unless the client has requested a capability indicating that
151 state should be maintained by the server. Clients MUST NOT require state
152 management on the server side in order to function correctly. This
153 permits simple round-robin load-balancing on the server side, without
154 needing to worry about state management.
155
156 agent
157 ~~~~~~~
158
159 The server can advertise the `agent` capability with a value `X` (in the
160 form `agent=X`) to notify the client that the server is running version
161 `X`. The client may optionally send its own agent string by including
162 the `agent` capability with a value `Y` (in the form `agent=Y`) in its
163 request to the server (but it MUST NOT do so if the server did not
164 advertise the agent capability). The `X` and `Y` strings may contain any
165 printable ASCII characters except space (i.e., the byte range 32 < x <
166 127), and are typically of the form "package/version" (e.g.,
167 "git/1.8.3.1"). The agent strings are purely informative for statistics
168 and debugging purposes, and MUST NOT be used to programmatically assume
169 the presence or absence of particular features.
170
171 ls-refs
172 ~~~~~~~~~
173
174 `ls-refs` is the command used to request a reference advertisement in v2.
175 Unlike the current reference advertisement, ls-refs takes in arguments
176 which can be used to limit the refs sent from the server.
177
178 Additional features not supported in the base command will be advertised
179 as the value of the command in the capability advertisement in the form
180 of a space separated list of features: "<command>=<feature 1> <feature 2>"
181
182 ls-refs takes in the following arguments:
183
184 symrefs
185 In addition to the object pointed by it, show the underlying ref
186 pointed by it when showing a symbolic ref.
187 peel
188 Show peeled tags.
189 ref-prefix <prefix>
190 When specified, only references having a prefix matching one of
191 the provided prefixes are displayed.
192
193 The output of ls-refs is as follows:
194
195 output = *ref
196 flush-pkt
197 ref = PKT-LINE(obj-id SP refname *(SP ref-attribute) LF)
198 ref-attribute = (symref | peeled)
199 symref = "symref-target:" symref-target
200 peeled = "peeled:" obj-id
201
202 fetch
203 ~~~~~~~
204
205 `fetch` is the command used to fetch a packfile in v2. It can be looked
206 at as a modified version of the v1 fetch where the ref-advertisement is
207 stripped out (since the `ls-refs` command fills that role) and the
208 message format is tweaked to eliminate redundancies and permit easy
209 addition of future extensions.
210
211 Additional features not supported in the base command will be advertised
212 as the value of the command in the capability advertisement in the form
213 of a space separated list of features: "<command>=<feature 1> <feature 2>"
214
215 A `fetch` request can take the following arguments:
216
217 want <oid>
218 Indicates to the server an object which the client wants to
219 retrieve. Wants can be anything and are not limited to
220 advertised objects.
221
222 have <oid>
223 Indicates to the server an object which the client has locally.
224 This allows the server to make a packfile which only contains
225 the objects that the client needs. Multiple 'have' lines can be
226 supplied.
227
228 done
229 Indicates to the server that negotiation should terminate (or
230 not even begin if performing a clone) and that the server should
231 use the information supplied in the request to construct the
232 packfile.
233
234 thin-pack
235 Request that a thin pack be sent, which is a pack with deltas
236 which reference base objects not contained within the pack (but
237 are known to exist at the receiving end). This can reduce the
238 network traffic significantly, but it requires the receiving end
239 to know how to "thicken" these packs by adding the missing bases
240 to the pack.
241
242 no-progress
243 Request that progress information that would normally be sent on
244 side-band channel 2, during the packfile transfer, should not be
245 sent. However, the side-band channel 3 is still used for error
246 responses.
247
248 include-tag
249 Request that annotated tags should be sent if the objects they
250 point to are being sent.
251
252 ofs-delta
253 Indicate that the client understands PACKv2 with delta referring
254 to its base by position in pack rather than by an oid. That is,
255 they can read OBJ_OFS_DELTA (ake type 6) in a packfile.
256
257 If the 'shallow' feature is advertised the following arguments can be
258 included in the clients request as well as the potential addition of the
259 'shallow-info' section in the server's response as explained below.
260
261 shallow <oid>
262 A client must notify the server of all commits for which it only
263 has shallow copies (meaning that it doesn't have the parents of
264 a commit) by supplying a 'shallow <oid>' line for each such
265 object so that the server is aware of the limitations of the
266 client's history. This is so that the server is aware that the
267 client may not have all objects reachable from such commits.
268
269 deepen <depth>
270 Requests that the fetch/clone should be shallow having a commit
271 depth of <depth> relative to the remote side.
272
273 deepen-relative
274 Requests that the semantics of the "deepen" command be changed
275 to indicate that the depth requested is relative to the client's
276 current shallow boundary, instead of relative to the requested
277 commits.
278
279 deepen-since <timestamp>
280 Requests that the shallow clone/fetch should be cut at a
281 specific time, instead of depth. Internally it's equivalent to
282 doing "git rev-list --max-age=<timestamp>". Cannot be used with
283 "deepen".
284
285 deepen-not <rev>
286 Requests that the shallow clone/fetch should be cut at a
287 specific revision specified by '<rev>', instead of a depth.
288 Internally it's equivalent of doing "git rev-list --not <rev>".
289 Cannot be used with "deepen", but can be used with
290 "deepen-since".
291
292 If the 'filter' feature is advertised, the following argument can be
293 included in the client's request:
294
295 filter <filter-spec>
296 Request that various objects from the packfile be omitted
297 using one of several filtering techniques. These are intended
298 for use with partial clone and partial fetch operations. See
299 `rev-list` for possible "filter-spec" values.
300
301 If the 'ref-in-want' feature is advertised, the following argument can
302 be included in the client's request as well as the potential addition of
303 the 'wanted-refs' section in the server's response as explained below.
304
305 want-ref <ref>
306 Indicates to the server that the client wants to retrieve a
307 particular ref, where <ref> is the full name of a ref on the
308 server.
309
310 The response of `fetch` is broken into a number of sections separated by
311 delimiter packets (0001), with each section beginning with its section
312 header.
313
314 output = *section
315 section = (acknowledgments | shallow-info | wanted-refs | packfile)
316 (flush-pkt | delim-pkt)
317
318 acknowledgments = PKT-LINE("acknowledgments" LF)
319 (nak | *ack)
320 (ready)
321 ready = PKT-LINE("ready" LF)
322 nak = PKT-LINE("NAK" LF)
323 ack = PKT-LINE("ACK" SP obj-id LF)
324
325 shallow-info = PKT-LINE("shallow-info" LF)
326 *PKT-LINE((shallow | unshallow) LF)
327 shallow = "shallow" SP obj-id
328 unshallow = "unshallow" SP obj-id
329
330 wanted-refs = PKT-LINE("wanted-refs" LF)
331 *PKT-LINE(wanted-ref LF)
332 wanted-ref = obj-id SP refname
333
334 packfile = PKT-LINE("packfile" LF)
335 *PKT-LINE(%x01-03 *%x00-ff)
336
337 acknowledgments section
338 * If the client determines that it is finished with negotiations
339 by sending a "done" line, the acknowledgments sections MUST be
340 omitted from the server's response.
341
342 * Always begins with the section header "acknowledgments"
343
344 * The server will respond with "NAK" if none of the object ids sent
345 as have lines were common.
346
347 * The server will respond with "ACK obj-id" for all of the
348 object ids sent as have lines which are common.
349
350 * A response cannot have both "ACK" lines as well as a "NAK"
351 line.
352
353 * The server will respond with a "ready" line indicating that
354 the server has found an acceptable common base and is ready to
355 make and send a packfile (which will be found in the packfile
356 section of the same response)
357
358 * If the server has found a suitable cut point and has decided
359 to send a "ready" line, then the server can decide to (as an
360 optimization) omit any "ACK" lines it would have sent during
361 its response. This is because the server will have already
362 determined the objects it plans to send to the client and no
363 further negotiation is needed.
364
365 shallow-info section
366 * If the client has requested a shallow fetch/clone, a shallow
367 client requests a fetch or the server is shallow then the
368 server's response may include a shallow-info section. The
369 shallow-info section will be included if (due to one of the
370 above conditions) the server needs to inform the client of any
371 shallow boundaries or adjustments to the clients already
372 existing shallow boundaries.
373
374 * Always begins with the section header "shallow-info"
375
376 * If a positive depth is requested, the server will compute the
377 set of commits which are no deeper than the desired depth.
378
379 * The server sends a "shallow obj-id" line for each commit whose
380 parents will not be sent in the following packfile.
381
382 * The server sends an "unshallow obj-id" line for each commit
383 which the client has indicated is shallow, but is no longer
384 shallow as a result of the fetch (due to its parents being
385 sent in the following packfile).
386
387 * The server MUST NOT send any "unshallow" lines for anything
388 which the client has not indicated was shallow as a part of
389 its request.
390
391 * This section is only included if a packfile section is also
392 included in the response.
393
394 wanted-refs section
395 * This section is only included if the client has requested a
396 ref using a 'want-ref' line and if a packfile section is also
397 included in the response.
398
399 * Always begins with the section header "wanted-refs".
400
401 * The server will send a ref listing ("<oid> <refname>") for
402 each reference requested using 'want-ref' lines.
403
404 * The server MUST NOT send any refs which were not requested
405 using 'want-ref' lines.
406
407 packfile section
408 * This section is only included if the client has sent 'want'
409 lines in its request and either requested that no more
410 negotiation be done by sending 'done' or if the server has
411 decided it has found a sufficient cut point to produce a
412 packfile.
413
414 * Always begins with the section header "packfile"
415
416 * The transmission of the packfile begins immediately after the
417 section header
418
419 * The data transfer of the packfile is always multiplexed, using
420 the same semantics of the 'side-band-64k' capability from
421 protocol version 1. This means that each packet, during the
422 packfile data stream, is made up of a leading 4-byte pkt-line
423 length (typical of the pkt-line format), followed by a 1-byte
424 stream code, followed by the actual data.
425
426 The stream code can be one of:
427 1 - pack data
428 2 - progress messages
429 3 - fatal error message just before stream aborts
430
431 server-option
432 ~~~~~~~~~~~~~~~
433
434 If advertised, indicates that any number of server specific options can be
435 included in a request. This is done by sending each option as a
436 "server-option=<option>" capability line in the capability-list section of
437 a request.
438
439 The provided options must not contain a NUL or LF character.