git/git.git
2 years agoMerge branch 'ab/commit-graph-write-progress'
Junio C Hamano [Tue, 5 Feb 2019 22:26:14 +0000 (14:26 -0800)]
Merge branch 'ab/commit-graph-write-progress'

The codepath to show progress meter while writing out commit-graph
file has been improved.

* ab/commit-graph-write-progress:
  commit-graph write: emit a percentage for all progress
  commit-graph write: add itermediate progress
  commit-graph write: remove empty line for readability
  commit-graph write: add more descriptive progress output
  commit-graph write: show progress for object search
  commit-graph write: more descriptive "writing out" output
  commit-graph write: add "Writing out" progress output
  commit-graph: don't call write_graph_chunk_extra_edges() unnecessarily
  commit-graph: rename "large edges" to "extra edges"

2 years agoMerge branch 'ab/commit-graph-write-optim'
Junio C Hamano [Tue, 5 Feb 2019 22:26:14 +0000 (14:26 -0800)]
Merge branch 'ab/commit-graph-write-optim'

The codepath to write out commit-graph has been optimized by
following the usual pattern of visiting objects in in-pack order.

* ab/commit-graph-write-optim:
  commit-graph write: use pack order when finding commits

2 years agoMerge branch 'js/t6042-timing-fix'
Junio C Hamano [Tue, 5 Feb 2019 22:26:14 +0000 (14:26 -0800)]
Merge branch 'js/t6042-timing-fix'

Test update.

* js/t6042-timing-fix:
  t6042: work around speed optimization on Windows

2 years agoMerge branch 'jk/add-ignore-errors-bit-assignment-fix'
Junio C Hamano [Tue, 5 Feb 2019 22:26:13 +0000 (14:26 -0800)]
Merge branch 'jk/add-ignore-errors-bit-assignment-fix'

"git add --ignore-errors" did not work as advertised and instead
worked as an unintended synonym for "git add --renormalize", which
has been fixed.

* jk/add-ignore-errors-bit-assignment-fix:
  add: use separate ADD_CACHE_RENORMALIZE flag

2 years agoMerge branch 'js/mingw-unc-path-w-backslashes'
Junio C Hamano [Tue, 5 Feb 2019 22:26:13 +0000 (14:26 -0800)]
Merge branch 'js/mingw-unc-path-w-backslashes'

In Git for Windows, "git clone \\server\share\path" etc. that uses
UNC paths from command line had bad interaction with its shell
emulation.

* js/mingw-unc-path-w-backslashes:
  mingw: special-case arguments to `sh`
  mingw (t5580): document bug when cloning from backslashed UNC paths

2 years agoMerge branch 'cc/test-ref-store-typofix'
Junio C Hamano [Tue, 5 Feb 2019 22:26:12 +0000 (14:26 -0800)]
Merge branch 'cc/test-ref-store-typofix'

An obvious typo in an assertion error message has been fixed.

* cc/test-ref-store-typofix:
  helper/test-ref-store: fix "new-sha1" vs "old-sha1" typo

2 years agoMerge branch 'jt/fetch-v2-sideband'
Junio C Hamano [Tue, 5 Feb 2019 22:26:11 +0000 (14:26 -0800)]
Merge branch 'jt/fetch-v2-sideband'

"git fetch" and "git upload-pack" learned to send all exchange over
the sideband channel while talking the v2 protocol.

* jt/fetch-v2-sideband:
  tests: define GIT_TEST_SIDEBAND_ALL
  {fetch,upload}-pack: sideband v2 fetch response
  sideband: reverse its dependency on pkt-line
  pkt-line: introduce struct packet_writer
  pack-protocol.txt: accept error packets in any context
  Use packet_reader instead of packet_read_line

2 years agoMerge branch 'sg/obstack-cast-function-type-fix'
Junio C Hamano [Tue, 5 Feb 2019 22:26:11 +0000 (14:26 -0800)]
Merge branch 'sg/obstack-cast-function-type-fix'

The compat/obstack code had casts that -Wcast-function-type
compilation option found questionable.

* sg/obstack-cast-function-type-fix:
  compat/obstack: fix -Wcast-function-type warnings

2 years agoMerge branch 'js/commit-graph-chunk-table-fix'
Junio C Hamano [Tue, 5 Feb 2019 22:26:11 +0000 (14:26 -0800)]
Merge branch 'js/commit-graph-chunk-table-fix'

The codepath to read from the commit-graph file attempted to read
past the end of it when the file's table-of-contents was corrupt.

* js/commit-graph-chunk-table-fix:
  Makefile: correct example fuzz build
  commit-graph: fix buffer read-overflow
  commit-graph, fuzz: add fuzzer for commit-graph

2 years agoMerge branch 'ld/git-p4-shelve-update-fix'
Junio C Hamano [Tue, 5 Feb 2019 22:26:10 +0000 (14:26 -0800)]
Merge branch 'ld/git-p4-shelve-update-fix'

"git p4" failed to update a shelved change when there were moved
files, which has been corrected.

* ld/git-p4-shelve-update-fix:
  git-p4: handle update of moved/copied files when updating a shelve
  git-p4: add failing test for shelved CL update involving move/copy

2 years agoMerge branch 'jt/get-reference-with-commit-graph'
Junio C Hamano [Tue, 5 Feb 2019 22:26:10 +0000 (14:26 -0800)]
Merge branch 'jt/get-reference-with-commit-graph'

Micro-optimize the code that prepares commit objects to be walked
by "git rev-list" when the commit-graph is available.

* jt/get-reference-with-commit-graph:
  revision: use commit graph in get_reference()

2 years agoMerge branch 'js/filter-options-should-use-plain-int'
Junio C Hamano [Tue, 5 Feb 2019 22:26:09 +0000 (14:26 -0800)]
Merge branch 'js/filter-options-should-use-plain-int'

Update the protocol message specification to allow only the limited
use of scaled quantities.  This is ensure potential compatibility
issues will not go out of hand.

* js/filter-options-should-use-plain-int:
  filter-options: expand scaled numbers
  tree:<depth>: skip some trees even when collecting omits
  list-objects-filter: teach tree:# how to handle >0

2 years agoMerge branch 'sb/more-repo-in-api'
Junio C Hamano [Tue, 5 Feb 2019 22:26:09 +0000 (14:26 -0800)]
Merge branch 'sb/more-repo-in-api'

The in-core repository instances are passed through more codepaths.

* sb/more-repo-in-api: (23 commits)
  t/helper/test-repository: celebrate independence from the_repository
  path.h: make REPO_GIT_PATH_FUNC repository agnostic
  commit: prepare free_commit_buffer and release_commit_memory for any repo
  commit-graph: convert remaining functions to handle any repo
  submodule: don't add submodule as odb for push
  submodule: use submodule repos for object lookup
  pretty: prepare format_commit_message to handle arbitrary repositories
  commit: prepare logmsg_reencode to handle arbitrary repositories
  commit: prepare repo_unuse_commit_buffer to handle any repo
  commit: prepare get_commit_buffer to handle any repo
  commit-reach: prepare in_merge_bases[_many] to handle any repo
  commit-reach: prepare get_merge_bases to handle any repo
  commit-reach.c: allow get_merge_bases_many_0 to handle any repo
  commit-reach.c: allow remove_redundant to handle any repo
  commit-reach.c: allow merge_bases_many to handle any repo
  commit-reach.c: allow paint_down_to_common to handle any repo
  commit: allow parse_commit* to handle any repo
  object: parse_object to honor its repository argument
  object-store: prepare has_{sha1, object}_file to handle any repo
  object-store: prepare read_object_file to deal with any repo
  ...

2 years agoFourth batch after 2.20
Junio C Hamano [Tue, 29 Jan 2019 20:54:55 +0000 (12:54 -0800)]
Fourth batch after 2.20

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoMerge branch 'it/log-format-source'
Junio C Hamano [Tue, 29 Jan 2019 20:47:56 +0000 (12:47 -0800)]
Merge branch 'it/log-format-source'

Custom userformat "log --format" learned %S atom that stands for
the tip the traversal reached the commit from, i.e. --source.

* it/log-format-source:
  log: add %S option (like --source) to log --format

2 years agoMerge branch 'js/add-e-clear-patch-before-stating'
Junio C Hamano [Tue, 29 Jan 2019 20:47:56 +0000 (12:47 -0800)]
Merge branch 'js/add-e-clear-patch-before-stating'

"git add -e" got confused when the change it wants to let the user
edit is smaller than the previous change that was left over in a
temporary file.

* js/add-e-clear-patch-before-stating:
  add --edit: truncate the patch file

2 years agoMerge branch 'bc/tree-walk-oid'
Junio C Hamano [Tue, 29 Jan 2019 20:47:56 +0000 (12:47 -0800)]
Merge branch 'bc/tree-walk-oid'

The code to walk tree objects has been taught that we may be
working with object names that are not computed with SHA-1.

* bc/tree-walk-oid:
  cache: make oidcpy always copy GIT_MAX_RAWSZ bytes
  tree-walk: store object_id in a separate member
  match-trees: use hashcpy to splice trees
  match-trees: compute buffer offset correctly when splicing
  tree-walk: copy object ID before use

2 years agoMerge branch 'jt/upload-pack-deepen-relative-proto-v2'
Junio C Hamano [Tue, 29 Jan 2019 20:47:55 +0000 (12:47 -0800)]
Merge branch 'jt/upload-pack-deepen-relative-proto-v2'

"git fetch --deepen=<more>" has been corrected to work over v2
protocol.

* jt/upload-pack-deepen-relative-proto-v2:
  upload-pack: teach deepen-relative in protocol v2
  fetch-pack: do not take shallow lock unnecessarily

2 years agoMerge branch 'jk/remote-insteadof-cleanup'
Junio C Hamano [Tue, 29 Jan 2019 20:47:55 +0000 (12:47 -0800)]
Merge branch 'jk/remote-insteadof-cleanup'

Code clean-up.

* jk/remote-insteadof-cleanup:
  remote: check config validity before creating rewrite struct

2 years agoMerge branch 'ms/http-no-more-failonerror'
Junio C Hamano [Tue, 29 Jan 2019 20:47:55 +0000 (12:47 -0800)]
Merge branch 'ms/http-no-more-failonerror'

Debugging help for http transport.

* ms/http-no-more-failonerror:
  test: test GIT_CURL_VERBOSE=1 shows an error
  remote-curl: unset CURLOPT_FAILONERROR
  remote-curl: define struct for CURLOPT_WRITEFUNCTION
  http: enable keep_error for HTTP requests
  http: support file handles for HTTP_KEEP_ERROR

2 years agoMerge branch 'os/rebase-runs-post-checkout-hook'
Junio C Hamano [Tue, 29 Jan 2019 20:47:55 +0000 (12:47 -0800)]
Merge branch 'os/rebase-runs-post-checkout-hook'

"git rebase" internally runs "checkout" to switch between branches,
and the command used to call the post-checkout hook, but the
reimplementation stopped doing so, which is getting fixed.

* os/rebase-runs-post-checkout-hook:
  rebase: run post-checkout hook on checkout
  t5403: simplify by using a single repository

2 years agoMerge branch 'bc/sha-256'
Junio C Hamano [Tue, 29 Jan 2019 20:47:55 +0000 (12:47 -0800)]
Merge branch 'bc/sha-256'

Add sha-256 hash and plug it through the code to allow building Git
with the "NewHash".

* bc/sha-256:
  hash: add an SHA-256 implementation using OpenSSL
  sha256: add an SHA-256 implementation using libgcrypt
  Add a base implementation of SHA-256 support
  commit-graph: convert to using the_hash_algo
  t/helper: add a test helper to compute hash speed
  sha1-file: add a constant for hash block size
  t: make the sha1 test-tool helper generic
  t: add basic tests for our SHA-1 implementation
  cache: make hashcmp and hasheq work with larger hashes
  hex: introduce functions to print arbitrary hashes
  sha1-file: provide functions to look up hash algorithms
  sha1-file: rename algorithm to "sha1"

2 years agoMerge branch 'sb/submodule-recursive-fetch-gets-the-tip'
Junio C Hamano [Tue, 29 Jan 2019 20:47:54 +0000 (12:47 -0800)]
Merge branch 'sb/submodule-recursive-fetch-gets-the-tip'

"git fetch --recurse-submodules" may not fetch the necessary commit
that is bound to the superproject, which is getting corrected.

* sb/submodule-recursive-fetch-gets-the-tip:
  fetch: ensure submodule objects fetched
  submodule.c: fetch in submodules git directory instead of in worktree
  submodule: migrate get_next_submodule to use repository structs
  repository: repo_submodule_init to take a submodule struct
  submodule: store OIDs in changed_submodule_names
  submodule.c: tighten scope of changed_submodule_names struct
  submodule.c: sort changed_submodule_names before searching it
  submodule.c: fix indentation
  sha1-array: provide oid_array_filter

2 years agoMerge branch 'jt/fetch-pack-v2'
Junio C Hamano [Tue, 29 Jan 2019 20:47:54 +0000 (12:47 -0800)]
Merge branch 'jt/fetch-pack-v2'

"git fetch-pack" now can talk the version 2 protocol.

* jt/fetch-pack-v2:
  fetch-pack: support protocol version 2

2 years agoMerge branch 'jk/proto-v2-hidden-refs-fix'
Junio C Hamano [Tue, 29 Jan 2019 20:47:54 +0000 (12:47 -0800)]
Merge branch 'jk/proto-v2-hidden-refs-fix'

The v2 upload-pack protocol implementation failed to honor
hidden-ref configuration, which has been corrected.
An earlier attempt reverted out of 'next'.

* jk/proto-v2-hidden-refs-fix:
  upload-pack: support hidden refs with protocol v2

2 years agoMerge branch 'jk/save-getenv-result'
Junio C Hamano [Tue, 29 Jan 2019 20:47:53 +0000 (12:47 -0800)]
Merge branch 'jk/save-getenv-result'

There were many places the code relied on the string returned from
getenv() to be non-volatile, which is not true, that have been
corrected.

* jk/save-getenv-result:
  builtin_diff(): read $GIT_DIFF_OPTS closer to use
  merge-recursive: copy $GITHEAD strings
  init: make a copy of $GIT_DIR string
  config: make a copy of $GIT_CONFIG string
  commit: copy saved getenv() result
  get_super_prefix(): copy getenv() result

2 years agoMerge branch 'pw/diff-color-moved-ws-fix'
Junio C Hamano [Tue, 29 Jan 2019 20:47:53 +0000 (12:47 -0800)]
Merge branch 'pw/diff-color-moved-ws-fix'

"git diff --color-moved-ws" updates.

* pw/diff-color-moved-ws-fix:
  diff --color-moved-ws: handle blank lines
  diff --color-moved-ws: modify allow-indentation-change
  diff --color-moved-ws: optimize allow-indentation-change
  diff --color-moved=zebra: be stricter with color alternation
  diff --color-moved-ws: fix false positives
  diff --color-moved-ws: demonstrate false positives
  diff: allow --no-color-moved-ws
  Use "whitespace" consistently
  diff: document --no-color-moved

2 years agoMerge branch 'ja/doc-build-l10n'
Junio C Hamano [Tue, 29 Jan 2019 20:47:53 +0000 (12:47 -0800)]
Merge branch 'ja/doc-build-l10n'

Prepare Documentation/Makefile so that manpage localization can
reuse it by overriding and tweaking the list of build products.

* ja/doc-build-l10n:
  Documentation/Makefile add optional targets for l10n

2 years agoMerge branch 'js/rebase-i-redo-exec'
Junio C Hamano [Tue, 29 Jan 2019 20:47:53 +0000 (12:47 -0800)]
Merge branch 'js/rebase-i-redo-exec'

"git rebase -i" learned to re-execute a command given with 'exec'
to run after it failed the last time.

* js/rebase-i-redo-exec:
  rebase: introduce a shortcut for --reschedule-failed-exec
  rebase: add a config option to default to --reschedule-failed-exec
  rebase: introduce --reschedule-failed-exec

2 years agoMerge branch 'cc/fetch-error-message-fix'
Junio C Hamano [Tue, 29 Jan 2019 20:47:53 +0000 (12:47 -0800)]
Merge branch 'cc/fetch-error-message-fix'

Error message fix.

* cc/fetch-error-message-fix:
  fetch: fix extensions.partialclone name in error message

2 years agoMerge branch 'cc/partial-clone-doc-typofix'
Junio C Hamano [Tue, 29 Jan 2019 20:47:52 +0000 (12:47 -0800)]
Merge branch 'cc/partial-clone-doc-typofix'

Doc fix.

* cc/partial-clone-doc-typofix:
  partial-clone: add missing 'is' in doc

2 years agoMerge branch 'kg/external-diff-save-env'
Junio C Hamano [Tue, 29 Jan 2019 20:47:51 +0000 (12:47 -0800)]
Merge branch 'kg/external-diff-save-env'

The code to drive GIT_EXTERNAL_DIFF command relied on the string
returned from getenv() to be non-volatile, which is not true, that
has been corrected.

* kg/external-diff-save-env:
  diff: ensure correct lifetime of external_diff_cmd

2 years agocommit-graph write: emit a percentage for all progress
Ævar Arnfjörð Bjarmason [Sat, 19 Jan 2019 20:21:21 +0000 (21:21 +0100)]
commit-graph write: emit a percentage for all progress

Follow-up 01ca387774 ("commit-graph: split up close_reachable()
progress output", 2018-11-19) by making the progress bars in
close_reachable() report a completion percentage. This fixes the last
occurrence where in the commit graph writing where we didn't report
that.

The change in 01ca387774 split up the 1x progress bar in
close_reachable() into 3x, but left them as dumb counters without a
percentage completion. Fixing that is easy, and the only reason it
wasn't done already is because that commit was rushed in during the
v2.20.0 RC period to fix the unrelated issue of over-reporting commit
numbers. See [1] and follow-ups for ML activity at the time and [2]
for an alternative approach where the progress bars weren't split up.

Now for e.g. linux.git we'll emit:

    $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write
    Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.
    Expanding reachable commits in commit graph: 100% (815990/815980), done.
    Computing commit graph generation numbers: 100% (815983/815983), done.
    Writing out commit graph in 4 passes: 100% (3263932/3263932), done.

1. https://public-inbox.org/git/20181119202300.18670-1-avarab@gmail.com/
2. https://public-inbox.org/git/20181122153922.16912-11-avarab@gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph write: add itermediate progress
Ævar Arnfjörð Bjarmason [Sat, 19 Jan 2019 20:21:20 +0000 (21:21 +0100)]
commit-graph write: add itermediate progress

Add progress output to sections of code between "Annotating[...]" and
"Computing[...]generation numbers". This can collectively take 5-10
seconds on a large enough repository.

On a test repository with I have with ~7 million commits and ~50
million objects we'll now emit:

    $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write
    Finding commits for commit graph among packed objects: 100% (124763727/124763727), done.
    Loading known commits in commit graph: 100% (18989461/18989461), done.
    Expanding reachable commits in commit graph: 100% (18989507/18989461), done.
    Clearing commit marks in commit graph: 100% (18989507/18989507), done.
    Counting distinct commits in commit graph: 100% (18989507/18989507), done.
    Finding extra edges in commit graph: 100% (18989507/18989507), done.
    Computing commit graph generation numbers: 100% (7250302/7250302), done.
    Writing out commit graph in 4 passes: 100% (29001208/29001208), done.

Whereas on a medium-sized repository such as linux.git these new
progress bars won't have time to kick in and as before and we'll still
emit output like:

    $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write
    Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.
    Expanding reachable commits in commit graph: 815990, done.
    Computing commit graph generation numbers: 100% (815983/815983), done.
    Writing out commit graph in 4 passes: 100% (3263932/3263932), done.

The "Counting distinct commits in commit graph" phase will spend most
of its time paused at "0/*" as we QSORT(...) the list. That's not
optimal, but at least we don't seem to be stalling anymore most of the
time.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph write: remove empty line for readability
Ævar Arnfjörð Bjarmason [Sat, 19 Jan 2019 20:21:19 +0000 (21:21 +0100)]
commit-graph write: remove empty line for readability

Remove the empty line between a QSORT(...) and the subsequent oideq()
for-loop. This makes it clearer that the QSORT(...) is being done so
that we can run the oideq() loop on adjacent OIDs. Amends code added
in 08fd81c9b6 ("commit-graph: implement write_commit_graph()",
2018-04-02).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph write: add more descriptive progress output
Ævar Arnfjörð Bjarmason [Sat, 19 Jan 2019 20:21:18 +0000 (21:21 +0100)]
commit-graph write: add more descriptive progress output

Make the progress output shown when we're searching for commits to
include in the graph more descriptive. This amends code I added in
7b0f229222 ("commit-graph write: add progress output", 2018-09-17).

Now, on linux.git, we'll emit this sort of output in the various modes
we support:

    $ git commit-graph write
    Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.
    [...]

    # Actually we don't emit this since this takes almost no time at
    # all. But if we did (s/_delayed//) we'd show:
    $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits
    Finding commits for commit graph from 630 refs: 100% (630/630), done.
    [...]

    $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack
    Finding commits for commit graph in 3 packs: 6529159, done.
    [...]

The middle on of those is going to be the output users might see in
practice, since it'll be emitted when they get the commit graph via
gc.writeCommitGraph=true. But as noted above you need a really large
number of refs for this message to show. It'll show up on a test
repository I have with ~165k refs:

    Finding commits for commit graph from 165203 refs: 100% (165203/165203), done.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph write: show progress for object search
Ævar Arnfjörð Bjarmason [Sat, 19 Jan 2019 20:21:17 +0000 (21:21 +0100)]
commit-graph write: show progress for object search

Show the percentage progress for the "Finding commits for commit
graph" phase for the common case where we're operating on all packs in
the repository, as "commit-graph write" or "gc" will do.

Before we'd emit on e.g. linux.git with "commit-graph write":

    Finding commits for commit graph: 6529159, done.
    [...]

And now:

    Finding commits for commit graph: 100% (6529159/6529159), done.
    [...]

Since the commit graph only includes those commits that are packed
(via for_each_packed_object(...)) the approximate_object_count()
returns the actual number of objects we're going to process.

Still, it is possible due to a race with "gc" or another process
maintaining packs that the number of objects we're going to process is
lower than what approximate_object_count() reported. In that case we
don't want to stop the progress bar short of 100%. So let's make sure
it snaps to 100% at the end.

The inverse case is also possible and more likely. I.e. that a new
pack has been added between approximate_object_count() and
for_each_packed_object(). In that case the percentage will go beyond
100%, and we'll do nothing to snap it back to 100% at the end.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph write: more descriptive "writing out" output
Ævar Arnfjörð Bjarmason [Sat, 19 Jan 2019 20:21:16 +0000 (21:21 +0100)]
commit-graph write: more descriptive "writing out" output

Make the "Writing out" part of the progress output more
descriptive. Depending on the shape of the graph we either make 3 or 4
passes over it.

Let's present this information to the user in case they're wondering
what this number, which is much larger than their number of commits,
has to do with writing out the commit graph. Now e.g. on linux.git we
emit:

    $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/linux commit-graph write
    Finding commits for commit graph: 6529159, done.
    Expanding reachable commits in commit graph: 815990, done.
    Computing commit graph generation numbers: 100% (815983/815983), done.
    Writing out commit graph in 4 passes: 100% (3263932/3263932), done.

A note on i18n: Why are we using the Q_() function and passing a
number & English text for a singular which'll never be used? Because
the plural rules of translated languages may not match those of
English, and to use the plural function we need to use this format.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph write: add "Writing out" progress output
Ævar Arnfjörð Bjarmason [Sat, 19 Jan 2019 20:21:15 +0000 (21:21 +0100)]
commit-graph write: add "Writing out" progress output

Add progress output to be shown when we're writing out the
commit-graph, this adds to the output already added in 7b0f229222
("commit-graph write: add progress output", 2018-09-17).

As noted in that commit most of the progress output isn't displayed on
small repositories, but before this change we'd noticeably hang for
2-3 seconds at the end on medium sized repositories such as linux.git.

Now we'll instead show output like this, and reduce the
human-observable times at which we're not producing progress output:

    $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/2015-04-03-1M-git commit-graph write
    Finding commits for commit graph: 13064614, done.
    Expanding reachable commits in commit graph: 1000447, done.
    Computing commit graph generation numbers: 100% (1000447/1000447), done.
    Writing out commit graph: 100% (3001341/3001341), done.

This "Writing out" number is 3x or 4x the number of commits, depending
on the graph we're processing. A later change will make this explicit
to the user.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: don't call write_graph_chunk_extra_edges() unnecessarily
SZEDER Gábor [Wed, 23 Jan 2019 17:51:22 +0000 (18:51 +0100)]
commit-graph: don't call write_graph_chunk_extra_edges() unnecessarily

The optional 'Extra Edge List' chunk of the commit graph file stores
parent information for commits with more than two parents.  Since the
chunk is optional, write_commit_graph() looks through all commits to
find those with more than two parents, and then writes the commit
graph file header accordingly, i.e. if there are no such commits, then
there won't be a 'Extra Edge List' chunk written, only the three
mandatory chunks.

However, when it later comes to writing actual chunk data,
write_commit_graph() unconditionally invokes
write_graph_chunk_extra_edges(), even when it was decided earlier that
that chunk won't be written.  Strictly speaking there is no bug here,
because write_graph_chunk_extra_edges() won't write anything if it
doesn't find any commits with more than two parents, but then it
unnecessarily and in vain looks through all commits once again in
search for such commits.

Don't call write_graph_chunk_extra_edges() when that chunk won't be
written to spare an unnecessary iteration over all commits.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: rename "large edges" to "extra edges"
SZEDER Gábor [Sat, 19 Jan 2019 20:21:13 +0000 (21:21 +0100)]
commit-graph: rename "large edges" to "extra edges"

The optional 'Large Edge List' chunk of the commit graph file stores
parent information for commits with more than two parents, and the
names of most of the macros, variables, struct fields, and functions
related to this chunk contain the term "large edges", e.g.
write_graph_chunk_large_edges().  However, it's not a really great
term, as the edges to the second and subsequent parents stored in this
chunk are not any larger than the edges to the first and second
parents stored in the "main" 'Commit Data' chunk.  It's the number of
edges, IOW number of parents, that is larger compared to non-merge and
"regular" two-parent merge commits.  And indeed, two functions in
'commit-graph.c' have a local variable called 'num_extra_edges' that
refer to the same thing, and this "extra edges" term is much better at
describing these edges.

So let's rename all these references to "large edges" in macro,
variable, function, etc. names to "extra edges".  There is a
GRAPH_OCTOPUS_EDGES_NEEDED macro as well; for the sake of consistency
rename it to GRAPH_EXTRA_EDGES_NEEDED.

We can do so safely without causing any incompatibility issues,
because the term "large edges" doesn't come up in the file format
itself in any form (the chunk's magic is {'E', 'D', 'G', 'E'}, there
is no 'L' in there), but only in the specification text.  The string
"large edges", however, does come up in the output of 'git
commit-graph read' and in tests looking at its input, but that command
is explicitly documented as debugging aid, so we can change its output
and the affected tests safely.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph write: use pack order when finding commits
Ævar Arnfjörð Bjarmason [Sat, 19 Jan 2019 20:21:12 +0000 (21:21 +0100)]
commit-graph write: use pack order when finding commits

Slightly optimize the "commit-graph write" step by using
FOR_EACH_OBJECT_PACK_ORDER with for_each_object_in_pack(). See commit
[1] and [2] for the facility and a similar optimization for "cat-file".

On Linux it is around 5% slower to run:

    echo 1 >/proc/sys/vm/drop_caches &&
    cat .git/objects/pack/* >/dev/null &&
    git cat-file --batch-all-objects --batch-check --unordered

Than the same thing with the "cat" omitted. This is as expected, since
we're iterating in pack order and the "cat" is extra work.

Before this change the opposite was true of "commit-graph write". We
were 6% faster if we first ran "cat" to efficiently populate the FS
cache for our sole big pack on linux.git, than if we had populated it
via for_each_object_in_pack(). Now we're 3% faster without the "cat"
instead.

My tests were done on an unloaded Linux 3.10 system with 10 runs for
each. Derrick Stolee did his own tests on Windows[3] showing a 2%
improvement with a high degree of accuracy.

1. 736eb88fdc ("for_each_packed_object: support iterating in
   pack-order", 2018-08-10)

2. 0750bb5b51 ("cat-file: support "unordered" output for
   --batch-all-objects", 2018-08-10)

3. https://public-inbox.org/git/f71fa868-25e8-a9c9-46a6-611b987f1a8f@gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoThird batch after 2.20 581/head
Junio C Hamano [Fri, 18 Jan 2019 21:56:54 +0000 (13:56 -0800)]
Third batch after 2.20

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoMerge branch 'js/gc-repack-close-before-remove'
Junio C Hamano [Fri, 18 Jan 2019 21:49:57 +0000 (13:49 -0800)]
Merge branch 'js/gc-repack-close-before-remove'

"git gc" and "git repack" did not close the open packfiles that
they found unneeded before removing them, which didn't work on a
platform incapable of removing an open file.  This has been
corrected.

* js/gc-repack-close-before-remove:
  gc/repack: release packs when needed

2 years agoMerge branch 'en/show-ref-doc-fix'
Junio C Hamano [Fri, 18 Jan 2019 21:49:56 +0000 (13:49 -0800)]
Merge branch 'en/show-ref-doc-fix'

Doc update.

* en/show-ref-doc-fix:
  git-show-ref.txt: fix order of flags

2 years agoMerge branch 'ot/ref-filter-object-info'
Junio C Hamano [Fri, 18 Jan 2019 21:49:56 +0000 (13:49 -0800)]
Merge branch 'ot/ref-filter-object-info'

The "--format=<placeholder>" option of for-each-ref, branch and tag
learned to show a few more traits of objects that can be learned by
the object_info API.

* ot/ref-filter-object-info:
  ref-filter: give uintmax_t to format with %PRIuMAX
  ref-filter: add docs for new options
  ref-filter: add tests for deltabase
  ref-filter: add deltabase option
  ref-filter: add tests for objectsize:disk
  ref-filter: add check for negative file size
  ref-filter: add objectsize:disk option

2 years agoMerge branch 'sg/stress-test'
Junio C Hamano [Fri, 18 Jan 2019 21:49:56 +0000 (13:49 -0800)]
Merge branch 'sg/stress-test'

Flaky tests can now be repeatedly run under load with the
"--stress" option.

* sg/stress-test:
  test-lib: add the '--stress' option to run a test repeatedly under load
  test-lib-functions: introduce the 'test_set_port' helper function
  test-lib: set $TRASH_DIRECTORY earlier
  test-lib: consolidate naming of test-results paths
  test-lib: parse command line options earlier
  test-lib: parse options in a for loop to keep $@ intact
  test-lib: extract Bash version check for '-x' tracing
  test-lib: translate SIGTERM and SIGHUP to an exit

2 years agoMerge branch 'rs/sha1-file-close-mapped-file-on-error'
Junio C Hamano [Fri, 18 Jan 2019 21:49:56 +0000 (13:49 -0800)]
Merge branch 'rs/sha1-file-close-mapped-file-on-error'

Code clean-up.

* rs/sha1-file-close-mapped-file-on-error:
  sha1-file: close fd of empty file in map_sha1_file_1()

2 years agoMerge branch 'rs/loose-object-cache-perffix'
Junio C Hamano [Fri, 18 Jan 2019 21:49:56 +0000 (13:49 -0800)]
Merge branch 'rs/loose-object-cache-perffix'

The loose object cache used to optimize existence look-up has been
updated.

* rs/loose-object-cache-perffix:
  object-store: retire odb_load_loose_cache()
  object-store: use one oid_array per subdirectory for loose cache
  object-store: factor out odb_clear_loose_cache()
  object-store: factor out odb_loose_cache()

2 years agoMerge branch 'po/git-p4-wo-login'
Junio C Hamano [Fri, 18 Jan 2019 21:49:55 +0000 (13:49 -0800)]
Merge branch 'po/git-p4-wo-login'

"git p4" update.

* po/git-p4-wo-login:
  git-p4: fix problem when p4 login is not necessary

2 years agoMerge branch 'mm/multimail-1.5'
Junio C Hamano [Fri, 18 Jan 2019 21:49:55 +0000 (13:49 -0800)]
Merge branch 'mm/multimail-1.5'

Update "git multimail" from the upstream.

* mm/multimail-1.5:
  git-multimail: update to release 1.5.0

2 years agoMerge branch 'tg/t5570-drop-racy-test'
Junio C Hamano [Fri, 18 Jan 2019 21:49:55 +0000 (13:49 -0800)]
Merge branch 'tg/t5570-drop-racy-test'

An inherently racy test that caused intermittent failures has been
removed.

* tg/t5570-drop-racy-test:
  Revert "t/lib-git-daemon: record daemon log"
  t5570: drop racy test

2 years agoMerge branch 'jk/dev-build-format-security'
Junio C Hamano [Fri, 18 Jan 2019 21:49:55 +0000 (13:49 -0800)]
Merge branch 'jk/dev-build-format-security'

Earlier we added "-Wformat-security" to developer builds, assuming
that "-Wall" (which includes "-Wformat" which in turn is required
to use "-Wformat-security") is always in effect.  This is not true
when config.mak.autogen is in use, unfortunately.  This has been
fixed by unconditionally adding "-Wall" to developer builds.

* jk/dev-build-format-security:
  config.mak.dev: add -Wall, primarily for -Wformat, to help autoconf users

2 years agoMerge branch 'so/cherry-pick-always-allow-m1'
Junio C Hamano [Fri, 18 Jan 2019 21:49:54 +0000 (13:49 -0800)]
Merge branch 'so/cherry-pick-always-allow-m1'

"git cherry-pick -m1" was forbidden when picking a non-merge
commit, even though there _is_ parent number 1 for such a commit.
This was done to avoid mistakes back when "cherry-pick" was about
picking a single commit, but is no longer useful with "cherry-pick"
that can pick a range of commits.  Now the "-m$num" option is
allowed when picking any commit, as long as $num names an existing
parent of the commit.

Technically this is a backward incompatible change; hopefully
nobody is relying on the error-checking behaviour.

* so/cherry-pick-always-allow-m1:
  t3506: validate '-m 1 -ff' is now accepted for non-merge commits
  t3502: validate '-m 1' argument is now accepted for non-merge commits
  cherry-pick: do not error on non-merge commits when '-m 1' is specified
  t3510: stop using '-m 1' to force failure mid-sequence of cherry-picks

2 years agoMerge branch 'nd/worktree-remove-with-uninitialized-submodules'
Junio C Hamano [Fri, 18 Jan 2019 21:49:54 +0000 (13:49 -0800)]
Merge branch 'nd/worktree-remove-with-uninitialized-submodules'

"git worktree remove" and "git worktree move" refused to work when
there is a submodule involved.  This has been loosened to ignore
uninitialized submodules.

* nd/worktree-remove-with-uninitialized-submodules:
  worktree: allow to (re)move worktrees with uninitialized submodules

2 years agoMerge branch 'sg/test-bash-version-fix'
Junio C Hamano [Fri, 18 Jan 2019 21:49:54 +0000 (13:49 -0800)]
Merge branch 'sg/test-bash-version-fix'

The test suite tried to see if it is run under bash, but the check
itself failed under some other implementations of shell (notably
under NetBSD).  This has been corrected.

* sg/test-bash-version-fix:
  test-lib: check Bash version for '-x' without using shell arrays

2 years agoMerge branch 'rb/hpe'
Junio C Hamano [Fri, 18 Jan 2019 21:49:54 +0000 (13:49 -0800)]
Merge branch 'rb/hpe'

Portability updates for the HPE NonStop platform.

* rb/hpe:
  compat/regex/regcomp.c: define intptr_t and uintptr_t on NonStop
  git-compat-util.h: add FLOSS headers for HPE NonStop
  config.mak.uname: support for modern HPE NonStop config.
  transport-helper: drop read/write errno checks
  transport-helper: use xread instead of read

2 years agoMerge branch 'ed/simplify-setup-git-dir'
Junio C Hamano [Fri, 18 Jan 2019 21:49:54 +0000 (13:49 -0800)]
Merge branch 'ed/simplify-setup-git-dir'

Code simplification.

* ed/simplify-setup-git-dir:
  Simplify handling of setup_git_directory_gently() failure cases.

2 years agoMerge branch 'cy/zsh-completion-SP-in-path'
Junio C Hamano [Fri, 18 Jan 2019 21:49:54 +0000 (13:49 -0800)]
Merge branch 'cy/zsh-completion-SP-in-path'

With zsh, "git cmd path<TAB>" was completed to "git cmd path name"
when the completed path has a special character like SP in it,
without any attempt to keep "path name" a single filename.  This
has been fixed to complete it to "git cmd path\ name" just like
Bash completion does.

* cy/zsh-completion-SP-in-path:
  completion: treat results of git ls-tree as file paths
  zsh: complete unquoted paths with spaces correctly

2 years agoMerge branch 'cy/completion-typofix'
Junio C Hamano [Fri, 18 Jan 2019 21:49:53 +0000 (13:49 -0800)]
Merge branch 'cy/completion-typofix'

Typofix.

* cy/completion-typofix:
  completion: fix typo in git-completion.bash

2 years agoMerge branch 'ew/ban-strncat'
Junio C Hamano [Fri, 18 Jan 2019 21:49:53 +0000 (13:49 -0800)]
Merge branch 'ew/ban-strncat'

The "strncat()" function is now among the banned functions.

* ew/ban-strncat:
  banned.h: mark strncat() as banned

2 years agoMerge branch 'ds/commit-graph-assert-missing-parents'
Junio C Hamano [Fri, 18 Jan 2019 21:49:53 +0000 (13:49 -0800)]
Merge branch 'ds/commit-graph-assert-missing-parents'

Tightening error checking in commit-graph writer.

* ds/commit-graph-assert-missing-parents:
  commit-graph: writing missing parents is a BUG

2 years agoMerge branch 'es/doc-worktree-guessremote-config'
Junio C Hamano [Fri, 18 Jan 2019 21:49:53 +0000 (13:49 -0800)]
Merge branch 'es/doc-worktree-guessremote-config'

Doc clarification.

* es/doc-worktree-guessremote-config:
  doc/config: do a better job of introducing 'worktree.guessRemote'

2 years agoMerge branch 'sb/submodule-unset-core-worktree-when-worktree-is-lost'
Junio C Hamano [Fri, 18 Jan 2019 21:49:53 +0000 (13:49 -0800)]
Merge branch 'sb/submodule-unset-core-worktree-when-worktree-is-lost'

The core.worktree setting in a submodule repository should not be
pointing at a directory when the submodule loses its working tree
(e.g. getting deinit'ed), but the code did not properly maintain
this invariant.

* sb/submodule-unset-core-worktree-when-worktree-is-lost:
  submodule deinit: unset core.worktree
  submodule--helper: fix BUG message in ensure_core_worktree
  submodule: unset core.worktree if no working tree is present
  submodule update: add regression test with old style setups

2 years agoMerge branch 'ma/asciidoctor'
Junio C Hamano [Fri, 18 Jan 2019 21:49:53 +0000 (13:49 -0800)]
Merge branch 'ma/asciidoctor'

Some of the documentation pages formatted incorrectly with
Asciidoctor, which have been fixed.

* ma/asciidoctor:
  git-status.txt: render tables correctly under Asciidoctor
  Documentation: do not nest open blocks
  git-column.txt: fix section header

2 years agoMerge branch 'jn/stripspace-wo-repository'
Junio C Hamano [Fri, 18 Jan 2019 21:49:53 +0000 (13:49 -0800)]
Merge branch 'jn/stripspace-wo-repository'

"git stripspace" should be usable outside a git repository, but
under the "-s" or "-c" mode, it didn't.

* jn/stripspace-wo-repository:
  stripspace: allow -s/-c outside git repository

2 years agoMerge branch 'sb/submodule-fetchjobs-default-to-one'
Junio C Hamano [Fri, 18 Jan 2019 21:49:52 +0000 (13:49 -0800)]
Merge branch 'sb/submodule-fetchjobs-default-to-one'

"git submodule update" ought to use a single job unless asked, but
by mistake used multiple jobs, which has been fixed.

* sb/submodule-fetchjobs-default-to-one:
  submodule update: run at most one fetch job unless otherwise set

2 years agoMerge branch 'la/quiltimport-keep-non-patch'
Junio C Hamano [Fri, 18 Jan 2019 21:49:52 +0000 (13:49 -0800)]
Merge branch 'la/quiltimport-keep-non-patch'

"git quiltimport" learned "--keep-non-patch" option.

* la/quiltimport-keep-non-patch:
  git-quiltimport: add --keep-non-patch option

2 years agoMerge branch 'nd/style-opening-brace'
Junio C Hamano [Fri, 18 Jan 2019 21:49:52 +0000 (13:49 -0800)]
Merge branch 'nd/style-opening-brace'

Code clean-up.

* nd/style-opening-brace:
  style: the opening '{' of a function is in a separate line

2 years agoMerge branch 'ds/gc-doc-typofix'
Junio C Hamano [Fri, 18 Jan 2019 21:49:52 +0000 (13:49 -0800)]
Merge branch 'ds/gc-doc-typofix'

Typofix.

* ds/gc-doc-typofix:
  git-gc.txt: fix typo about gc.writeCommitGraph

2 years agomingw: special-case arguments to `sh`
Johannes Schindelin [Thu, 17 Jan 2019 20:14:48 +0000 (12:14 -0800)]
mingw: special-case arguments to `sh`

The MSYS2 runtime does its best to emulate the command-line wildcard
expansion and de-quoting which would be performed by the calling Unix
shell on Unix systems.

Those Unix shell quoting rules differ from the quoting rules applying to
Windows' cmd and Powershell, making it a little awkward to quote
command-line parameters properly when spawning other processes.

In particular, git.exe passes arguments to subprocesses that are *not*
intended to be interpreted as wildcards, and if they contain
backslashes, those are not to be interpreted as escape characters, e.g.
when passing Windows paths.

Note: this is only a problem when calling MSYS2 executables, not when
calling MINGW executables such as git.exe. However, we do call MSYS2
executables frequently, most notably when setting the use_shell flag in
the child_process structure.

There is no elegant way to determine whether the .exe file to be
executed is an MSYS2 program or a MINGW one. But since the use case of
passing a command line through the shell is so prevalent, we need to
work around this issue at least when executing sh.exe.

Let's introduce an ugly, hard-coded test whether argv[0] is "sh", and
whether it refers to the MSYS2 Bash, to determine whether we need to
quote the arguments differently than usual.

That still does not fix the issue completely, but at least it is
something.

Incidentally, this also fixes the problem where `git clone \\server\repo`
failed due to incorrect handling of the backslashes when handing the path
to the git-upload-pack process.

Further, we need to take care to quote not only whitespace and
backslashes, but also curly brackets. As aliases frequently go through
the MSYS2 Bash, and as aliases frequently get parameters such as
HEAD@{yesterday}, this is really important. As an early version of this
patch broke this, let's make sure that this does not regress by adding a
test case for that.

Helped-by: Kim Gybels <kgybels@infogroep.be>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agomingw (t5580): document bug when cloning from backslashed UNC paths
Johannes Schindelin [Thu, 17 Jan 2019 20:14:47 +0000 (12:14 -0800)]
mingw (t5580): document bug when cloning from backslashed UNC paths

Due to a quirk in Git's method to spawn git-upload-pack, there is a
problem when passing paths with backslashes in them: Git will force the
command-line through the shell, which has different quoting semantics in
Git for Windows (being an MSYS2 program) than regular Win32 executables
such as git.exe itself.

The symptom is that the first of the two backslashes in UNC paths of the
form \\myserver\folder\repository.git is *stripped off*.

Document this bug by introducing a test case.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agogit-p4: handle update of moved/copied files when updating a shelve
Luke Diamand [Fri, 18 Jan 2019 09:36:56 +0000 (09:36 +0000)]
git-p4: handle update of moved/copied files when updating a shelve

Perforce requires a complete list of files being operated on. If
git is updating an existing shelved changelist, then any files
which are moved or copied were not being added to this list.

Signed-off-by: Luke Diamand <luke@diamand.org>
Acked-by: Andrey Mazo <amazo@checkvideo.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agogit-p4: add failing test for shelved CL update involving move/copy
Luke Diamand [Fri, 18 Jan 2019 09:36:55 +0000 (09:36 +0000)]
git-p4: add failing test for shelved CL update involving move/copy

Updating a shelved P4 changelist where one or more files have
been moved or copied does not work. Add a test for this.

The problem is that P4 requires a complete list of the files being
changed, and move/copy only includes the _source_ in the case of
updating a shelved changelist. This results in errors from Perforce
such as:

  //depot/src - needs tofile //depot/dst
  Submit aborted -- fix problems then use 'p4 submit -c 1234'

Signed-off-by: Luke Diamand <luke@diamand.org>
Acked-by: Andrey Mazo <amazo@checkvideo.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoadd: use separate ADD_CACHE_RENORMALIZE flag
Jeff King [Thu, 17 Jan 2019 16:27:11 +0000 (11:27 -0500)]
add: use separate ADD_CACHE_RENORMALIZE flag

Commit 9472935d81 (add: introduce "--renormalize", 2017-11-16) taught
git-add to pass HASH_RENORMALIZE to add_to_index(), which then passes
the flag along to index_path(). However, the flags taken by
add_to_index() and the ones taken by index_path() are distinct
namespaces. We cannot take HASH_* flags in add_to_index(), because they
overlap with the ADD_CACHE_* flags we already take (in this case,
HASH_RENORMALIZE conflicts with ADD_CACHE_IGNORE_ERRORS).

We can solve this by adding a new ADD_CACHE_RENORMALIZE flag, and using
it to set HASH_RENORMALIZE within add_to_index(). In order to make it
clear that these two flags come from distinct sets, let's also change
the name "newflags" in the function to "hash_flags".

Reported-by: Dmitriy Smirnov <dmitriy.smirnov@jetbrains.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agot6042: work around speed optimization on Windows
Johannes Schindelin [Thu, 17 Jan 2019 08:29:21 +0000 (00:29 -0800)]
t6042: work around speed optimization on Windows

When Git determines whether a file has changed, it looks at the mtime,
at the file size, and to detect changes even if the mtime is the same
(on Windows, the mtime granularity is 100ns, read: if two files are
written within the same 100ns time slot, they have the same mtime) and
even if the file size is the same, Git also looks at the inode/device
numbers.

This design obviously comes from a Linux background, where `lstat()`
calls were designed to be cheap.

On Windows, there is no `lstat()`. It has to be emulated. And while
obtaining the mtime and the file size is not all that expensive (you can
get both with a single `GetFileAttributesW()` call), obtaining the
equivalent of the inode and device numbers is very expensive (it
requires a call to `GetFileInformationByHandle()`, which in turn
requires a file handle, which is *a lot* more expensive than one might
imagine).

As it is very uncommon for developers to modify files within 100ns time
slots, Git for Windows chooses not to fill inode/device numbers
properly, but simply sets them to 0.

However, in t6042 the files file_v1 and file_v2 are typically written
within the same 100ns time slot, and they do not differ in file size. So
the minor modification is not picked up.

Let's work around this issue by avoiding the `git mv` calls in the
'mod6-setup: chains of rename/rename(1to2) and rename/rename(2to1)' test
case. The target files are overwritten anyway, so it is not like we
really rename those files. This fixes the issue because `git add` will
now add the files as new files (as opposed to existing, just renamed
files).

Functionally, we do not change anything because we replace two `git mv
<old> <new>` calls (where `<new>` is completely overwritten and `git
add`ed later anyway) by `git rm <old>` calls (removing other files, too,
that are also completely overwritten and `git add`ed later).

Reviewed-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotests: define GIT_TEST_SIDEBAND_ALL
Jonathan Tan [Wed, 16 Jan 2019 19:28:15 +0000 (11:28 -0800)]
tests: define GIT_TEST_SIDEBAND_ALL

Define a GIT_TEST_SIDEBAND_ALL environment variable meant to be used
from tests. When set to true, this overrides uploadpack.allowsidebandall
to true, allowing the entire test suite to be run as if this
configuration is in place for all repositories.

As of this patch, all tests pass whether GIT_TEST_SIDEBAND_ALL is unset
or set to 1.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years ago{fetch,upload}-pack: sideband v2 fetch response
Jonathan Tan [Wed, 16 Jan 2019 19:28:14 +0000 (11:28 -0800)]
{fetch,upload}-pack: sideband v2 fetch response

Currently, a response to a fetch request has sideband support only while
the packfile is being sent, meaning that the server cannot send notices
until the start of the packfile.

Extend sideband support in protocol v2 fetch responses to the whole
response. upload-pack will advertise it if the
uploadpack.allowsidebandall configuration variable is set, and
fetch-pack will automatically request it if advertised.

If the sideband is to be used throughout the whole response, upload-pack
will use it to send errors instead of prefixing a PKT-LINE payload with
"ERR ".

This will be tested in a subsequent patch.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agosideband: reverse its dependency on pkt-line
Jonathan Tan [Wed, 16 Jan 2019 19:28:13 +0000 (11:28 -0800)]
sideband: reverse its dependency on pkt-line

A subsequent patch will teach struct packet_reader a new field that, if
set, instructs it to interpret read data as multiplexed. This will
create a dependency from pkt-line to sideband.

To avoid a circular dependency, split recv_sideband() into 2 parts: the
reading loop (left in recv_sideband()) and the processing of the
contents (in demultiplex_sideband()), and move the former into pkt-line.
This reverses the direction of dependency: sideband no longer depends on
pkt-line, and pkt-line now depends on sideband.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocompat/obstack: fix -Wcast-function-type warnings
SZEDER Gábor [Thu, 17 Jan 2019 01:29:09 +0000 (02:29 +0100)]
compat/obstack: fix -Wcast-function-type warnings

GCC 8 introduced the new -Wcast-function-type warning, which is
implied by -Wextra (which, in turn is enabled in our DEVELOPER flags).
When building Git with GCC 8 and this warning enabled on a non-glibc
platform [1], one is greeted with a screenful of compiler
warnings/errors:

  compat/obstack.c: In function '_obstack_begin':
  compat/obstack.c:162:17: error: cast between incompatible function types from 'void * (*)(long int)' to 'struct _obstack_chunk * (*)(void *, long int)' [-Werror=cast-function-type]
     h->chunkfun = (struct _obstack_chunk * (*)(void *, long)) chunkfun;
                   ^
  compat/obstack.c:163:16: error: cast between incompatible function types from 'void (*)(void *)' to 'void (*)(void *, struct _obstack_chunk *)' [-Werror=cast-function-type]
     h->freefun = (void (*) (void *, struct _obstack_chunk *)) freefun;
                  ^
  compat/obstack.c:116:8: error: cast between incompatible function types from 'struct _obstack_chunk * (*)(void *, long int)' to 'struct _obstack_chunk * (*)(long int)' [-Werror=cast-function-type]
      : (*(struct _obstack_chunk *(*) (long)) (h)->chunkfun) ((size)))
          ^
  compat/obstack.c:168:22: note: in expansion of macro 'CALL_CHUNKFUN'
     chunk = h->chunk = CALL_CHUNKFUN (h, h -> chunk_size);
                        ^~~~~~~~~~~~~
  <snip>

'struct obstack' stores pointers to two functions to allocate and free
"chunks", and depending on how obstack is used, these functions take
either one parameter (like standard malloc() and free() do; this is
how we use it in 'kwset.c') or two parameters.  Presumably to reduce
memory footprint, a single field is used to store the function pointer
for both signatures, and then it's casted to the appropriate signature
when the function pointer is accessed.  These casts between function
pointers with different number of parameters are what trigger those
compiler errors.

Modify 'struct obstack' to use unions to store function pointers with
different signatures, and then use the union member with the
appropriate signature when accessing these function pointers.  This
eliminates the need for those casts, and thus avoids this compiler
error.

[1] Compiling 'compat/obstack.c' on a platform with glibc is sort of
    a noop, see the comment before '#  define ELIDE_CODE', so this is
    not an issue on common Linux distros.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agohelper/test-ref-store: fix "new-sha1" vs "old-sha1" typo
Christian Couder [Sun, 6 Jan 2019 15:46:37 +0000 (16:46 +0100)]
helper/test-ref-store: fix "new-sha1" vs "old-sha1" typo

It looks like it is a copy-paste error  made in 80f2a6097c
(t/helper: add test-ref-store to test ref-store functions,
2017-03-26) to pass "old-sha1" instead of "new-sha1" to
notnull() when we get the new sha1 argument from
const char **argv.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
Acked-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoMakefile: correct example fuzz build
Josh Steadmon [Tue, 15 Jan 2019 22:25:52 +0000 (14:25 -0800)]
Makefile: correct example fuzz build

The comment explaining how to build the fuzzers was broken in
927c77e7d4d ("Makefile: use FUZZ_CXXFLAGS for linking fuzzers",
2018-11-14).

When building fuzzers, all .c files must be compiled with coverage
tracing enabled. This is not possible when using only FUZZ_CXXFLAGS, as
that flag is only applied to the fuzzers themselves. Switching back to
CFLAGS fixes the issue.

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: fix buffer read-overflow
Josh Steadmon [Tue, 15 Jan 2019 22:25:51 +0000 (14:25 -0800)]
commit-graph: fix buffer read-overflow

fuzz-commit-graph identified a case where Git will read past the end of
a buffer containing a commit graph if the graph's header has an
incorrect chunk count. A simple bounds check in parse_commit_graph()
prevents this.

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph, fuzz: add fuzzer for commit-graph
Josh Steadmon [Tue, 15 Jan 2019 22:25:50 +0000 (14:25 -0800)]
commit-graph, fuzz: add fuzzer for commit-graph

Break load_commit_graph_one() into a new function, parse_commit_graph().
The latter function operates on arbitrary buffers, which makes it
suitable as a fuzzing target. Since parse_commit_graph() is only called
by load_commit_graph_one() (and the fuzzer described below), we omit
error messages that would be duplicated by the caller.

Adds fuzz-commit-graph.c, which provides a fuzzing entry point
compatible with libFuzzer (and possibly other fuzzing engines).

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agofilter-options: expand scaled numbers
Josh Steadmon [Tue, 8 Jan 2019 00:17:09 +0000 (16:17 -0800)]
filter-options: expand scaled numbers

When communicating with a remote server or a subprocess, use
expanded numbers rather than numbers with scaling suffix in the
object filter spec (e.g.  "limit:blob=1k" becomes
"limit:blob=1024").

Update the protocol docs to note that clients should always perform this
expansion, to allow for more compatibility between server
implementations.

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotree:<depth>: skip some trees even when collecting omits
Matthew DeVore [Wed, 9 Jan 2019 02:59:14 +0000 (18:59 -0800)]
tree:<depth>: skip some trees even when collecting omits

If a tree has already been recorded as omitted, we don't need to
traverse it again just to collect its omits. Stop traversing trees a
second time when collecting omits.

Signed-off-by: Matthew DeVore <matvore@google.com>
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agolist-objects-filter: teach tree:# how to handle >0
Matthew DeVore [Wed, 9 Jan 2019 02:59:13 +0000 (18:59 -0800)]
list-objects-filter: teach tree:# how to handle >0

Implement positive values for <depth> in the tree:<depth> filter. The
exact semantics are described in Documentation/rev-list-options.txt.

The long-term goal at the end of this is to allow a partial clone to
eagerly fetch an entire directory of files by fetching a tree and
specifying <depth>=1. This, for instance, would make a build operation
fast and convenient. It is fast because the partial clone does not need
to fetch each file individually, and convenient because the user does
not need to supply a sparse-checkout specification.

Another way of considering this feature is as a way to reduce
round-trips, since the client can get any number of levels of
directories in a single request, rather than wait for each level of tree
objects to come back, whose entries are used to construct a new request.

Signed-off-by: Matthew DeVore <matvore@google.com>
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoMerge branch 'nd/the-index' into md/list-objects-filter-by-depth
Junio C Hamano [Fri, 28 Dec 2018 18:41:41 +0000 (10:41 -0800)]
Merge branch 'nd/the-index' into md/list-objects-filter-by-depth

2 years agopkt-line: introduce struct packet_writer
Jonathan Tan [Tue, 15 Jan 2019 19:40:27 +0000 (11:40 -0800)]
pkt-line: introduce struct packet_writer

A future patch will allow the client to request multiplexing of the
entire fetch response (and not only during packfile transmission), which
in turn allows the server to send progress and keepalive messages at any
time during the response.

It will be convenient for a future patch if writing options
(specifically, whether the written data is to be multiplexed) could be
controlled from a single place, so create struct packet_writer to serve
as that place, and modify upload-pack to use it.

Currently, it only stores the output fd, but a subsequent patch will (as
described above) introduce an option to determine if the written data is
to be multiplexed.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoadd --edit: truncate the patch file
Johannes Schindelin [Tue, 15 Jan 2019 15:42:52 +0000 (07:42 -0800)]
add --edit: truncate the patch file

If there is already a .git/ADD_EDIT.patch file, we fail to truncate it
properly, which could result in very funny errors.

Of course, this file should not be left lying around. But at least in
one case, there was a stale copy, larger than the current diff. So the
result was a corrupt diff.

Let's just truncate the file when we write it and not worry about it too
much.

Reported by J Wyman.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocache: make oidcpy always copy GIT_MAX_RAWSZ bytes
brian m. carlson [Tue, 15 Jan 2019 00:39:45 +0000 (00:39 +0000)]
cache: make oidcpy always copy GIT_MAX_RAWSZ bytes

There are some situations in which we want to store an object ID into
struct object_id without the_hash_algo necessarily being set correctly.
One such case is when cloning a repository, where we must read refs from
the remote side without having a repository from which to read the
preferred algorithm.

In this cases, we may have the_hash_algo set to SHA-1, which is the
default, but read refs into struct object_id that are SHA-256. When
copying these values, we will want to copy them completely, not just the
first 20 bytes. Consequently, make sure that oidcpy copies the maximum
number of bytes at all times, regardless of the setting of
the_hash_algo.

Since oidcpy and hashcpy are no longer functionally identical, remove
the Cocinelle object_id transformations that convert from one into the
other.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotree-walk: store object_id in a separate member
brian m. carlson [Tue, 15 Jan 2019 00:39:44 +0000 (00:39 +0000)]
tree-walk: store object_id in a separate member

When parsing a tree, we read the object ID directly out of the tree
buffer. This is normally fine, but such an object ID cannot be used with
oidcpy, which copies GIT_MAX_RAWSZ bytes, because if we are using SHA-1,
there may not be that many bytes to copy.

Instead, store the object ID in a separate struct member. Since we can
no longer efficiently compute the path length, store that information as
well in struct name_entry. Ensure we only copy the object ID into the
new buffer if the path length is nonzero, as some callers will pass us
an empty path with no object ID following it, and we will not want to
read past the end of the buffer.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agomatch-trees: use hashcpy to splice trees
brian m. carlson [Tue, 15 Jan 2019 00:39:43 +0000 (00:39 +0000)]
match-trees: use hashcpy to splice trees

When we splice trees together, we operate in place on the tree buffer.
If we're using SHA-1 for the hash algorithm, we may not have a full
GIT_MAX_RAWSZ (32) bytes to copy. Consequently, it doesn't logically
make sense for us to use a struct object_id to represent this type,
since it isn't a complete object.

Represent this value as a unsigned char pointer instead and copy it when
necessary.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agomatch-trees: compute buffer offset correctly when splicing
brian m. carlson [Tue, 15 Jan 2019 00:39:42 +0000 (00:39 +0000)]
match-trees: compute buffer offset correctly when splicing

Currently, the struct object_id pointer returned from tree_entry_extract
lives directly inside the parsed tree buffer. In a future commit, this
will change so that it instead points to a dedicated struct member.
Since in this code path, we want to modify the buffer directly, compute
the buffer offset we want to modify by using the pointer to the path
instead.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotree-walk: copy object ID before use
brian m. carlson [Tue, 15 Jan 2019 00:39:41 +0000 (00:39 +0000)]
tree-walk: copy object ID before use

In a future commit, the pointer returned by tree_entry_extract will
point into the struct tree_desc, causing its lifetime to be bound to
that of the struct tree_desc itself.  To ensure this code path keeps
working, copy the object_id into a local variable so that it lives long
enough.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoSecond batch after 2.20
Junio C Hamano [Mon, 14 Jan 2019 23:33:36 +0000 (15:33 -0800)]
Second batch after 2.20

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoMerge branch 'do/gitweb-strict-export-conf-doc'
Junio C Hamano [Mon, 14 Jan 2019 23:29:33 +0000 (15:29 -0800)]
Merge branch 'do/gitweb-strict-export-conf-doc'

Doc update.

* do/gitweb-strict-export-conf-doc:
  docs: fix $strict_export text in gitweb.conf.txt

2 years agoMerge branch 'nd/indentation-fix'
Junio C Hamano [Mon, 14 Jan 2019 23:29:32 +0000 (15:29 -0800)]
Merge branch 'nd/indentation-fix'

Code cleanup.

* nd/indentation-fix:
  Indent code with TABs

2 years agoMerge branch 'en/directory-renames-nothanks-doc-update'
Junio C Hamano [Mon, 14 Jan 2019 23:29:32 +0000 (15:29 -0800)]
Merge branch 'en/directory-renames-nothanks-doc-update'

Doc update.

* en/directory-renames-nothanks-doc-update:
  git-rebase.txt: update note about directory rename detection and am

2 years agoMerge branch 'bw/mailmap'
Junio C Hamano [Mon, 14 Jan 2019 23:29:32 +0000 (15:29 -0800)]
Merge branch 'bw/mailmap'

* bw/mailmap:
  mailmap: update brandon williams's email address