git/git.git
2 years agocommit-reach: fix memory and flag leaks
Derrick Stolee [Fri, 21 Sep 2018 15:05:27 +0000 (08:05 -0700)]
commit-reach: fix memory and flag leaks

The can_all_from_reach_with_flag() method uses 'assign_flag' as a
value we can use to mark objects temporarily during our commit walk.
The intent is that these flags are removed from all objects before
returning. However, this is not the case.

The 'from' array could also contain objects that are not commits, and
we mark those objects with 'assign_flag'. Add a loop to the 'cleanup'
section that removes these markers.

Also, we forgot to free() the memory for 'list', so add that to the
'cleanup' section.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-reach: properly peel tags
Derrick Stolee [Fri, 21 Sep 2018 15:05:26 +0000 (08:05 -0700)]
commit-reach: properly peel tags

The can_all_from_reach_with_flag() algorithm was refactored in 4fbcca4e
"commit-reach: make can_all_from_reach... linear" but incorrectly
assumed that all objects provided were commits. During a fetch
negotiation, ok_to_give_up() in upload-pack.c may provide unpeeled tags
to the 'from' array. The current code creates a segfault.

Add a direct call to can_all_from_reach_with_flag() in 'test-tool reach'
and add a test in t6600-test-reach.sh that demonstrates this segfault.

Correct the issue by peeling tags when investigating the initial list
of objects in the 'from' array.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-reach: correct accidental #include of C file
Jonathan Nieder [Tue, 28 Aug 2018 21:36:57 +0000 (14:36 -0700)]
commit-reach: correct accidental #include of C file

Without this change, the build breaks with clang:

 libgit/ref-filter.pic.o: multiple definition of 'filter_refs'
 libgit/commit-reach.pic.o: previous definition here

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-reach: use can_all_from_reach
Derrick Stolee [Fri, 20 Jul 2018 16:33:30 +0000 (16:33 +0000)]
commit-reach: use can_all_from_reach

The is_descendant_of method previously used in_merge_bases() to check if
the commit can reach any of the commits in the provided list. This had
two performance problems:

1. The performance is quadratic in worst-case.

2. A single in_merge_bases() call requires walking beyond the target
   commit in order to find the full set of boundary commits that may be
   merge-bases.

The can_all_from_reach method avoids this quadratic behavior and can
limit the search beyond the target commits using generation numbers. It
requires a small prototype adjustment to stop using commit-date as a
cutoff, as that optimization is no longer appropriate here.

Since in_merge_bases() uses paint_down_to_common(), is_descendant_of()
naturally found cutoffs to avoid walking the entire commit graph. Since
we want to always return the correct result, we cannot use the
min_commit_date cutoff in can_all_from_reach. We then rely on generation
numbers to provide the cutoff.

Since not all repos will have a commit-graph file, nor will we always
have generation numbers computed for a commit-graph file, create a new
method, generation_numbers_enabled(), that checks for a commit-graph
file and sees if the first commit in the file has a non-zero generation
number. In the case that we do not have generation numbers, use the old
logic for is_descendant_of().

Performance was meausured on a copy of the Linux repository using the
'test-tool reach is_descendant_of' command using this input:

A:v4.9
X:v4.10
X:v4.11
X:v4.12
X:v4.13
X:v4.14
X:v4.15
X:v4.16
X:v4.17
X.v3.0

Note that this input is tailored to demonstrate the quadratic nature of
the previous method, as it will compute merge-bases for v4.9 versus all
of the later versions before checking against v4.1.

Before: 0.26 s
 After: 0.21 s

Since we previously used the is_descendant_of method in the ref_newer
method, we also measured performance there using
'test-tool reach ref_newer' with this input:

A:v4.9
B:v3.19

Before: 0.10 s
 After: 0.08 s

By adding a new commit with parent v3.19, we test the non-reachable case
of ref_newer:

Before: 0.09 s
 After: 0.08 s

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-reach: make can_all_from_reach... linear
Derrick Stolee [Fri, 20 Jul 2018 16:33:28 +0000 (16:33 +0000)]
commit-reach: make can_all_from_reach... linear

The can_all_from_reach_with_flags() algorithm is currently quadratic in
the worst case, because it calls the reachable() method for every 'from'
without tracking which commits have already been walked or which can
already reach a commit in 'to'.

Rewrite the algorithm to walk each commit a constant number of times.

We also add some optimizations that should work for the main consumer of
this method: fetch negotitation (haves/wants).

The first step includes using a depth-first-search (DFS) from each
'from' commit, sorted by ascending generation number. We do not walk
beyond the minimum generation number or the minimum commit date. This
DFS is likely to be faster than the existing reachable() method because
we expect previous ref values to be along the first-parent history.

If we find a target commit, then we mark everything in the DFS stack as
a RESULT. This expands the set of targets for the other 'from' commits.
We also mark the visited commits using 'assign_flag' to prevent re-
walking the same commits.

We still need to clear our flags at the end, which is why we will have a
total of three visits to each commit.

Performance was measured on the Linux repository using
'test-tool reach can_all_from_reach'. The input included rows seeded by
tag values. The "small" case included X-rows as v4.[0-9]* and Y-rows as
v3.[0-9]*. This mimics a (very large) fetch that says "I have all major
v3 releases and want all major v4 releases." The "large" case included
X-rows as "v4.*" and Y-rows as "v3.*". This adds all release-candidate
tags to the set, which does not greatly increase the number of objects
that are considered, but does increase the number of 'from' commits,
demonstrating the quadratic nature of the previous code.

Small Case:

Before: 1.52 s
 After: 0.26 s

Large Case:

Before: 3.50 s
 After: 0.27 s

Note how the time increases between the two cases in the two versions.
The new code increases relative to the number of commits that need to be
walked, but not directly relative to the number of 'from' commits.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-reach: replace ref_newer logic
Derrick Stolee [Fri, 20 Jul 2018 16:33:27 +0000 (16:33 +0000)]
commit-reach: replace ref_newer logic

The ref_newer method is used by 'git push' to check if a force-push is
required. This method does not use any kind of cutoff when walking, so
in the case of a force-push will walk all reachable commits.

The is_descendant_of method already uses paint_down_to_common along with
cutoffs. By translating the ref_newer arguments into the commit and
commit_list required by is_descendant_of, we can have one fewer commit
walk and also improve our performance!

For a copy of the Linux repository, 'test-tool reach ref_newer' presents
the following improvements with the specified input. In the case that
ref_newer returns 1, there is no improvement. The improvement is in the
second case where ref_newer returns 0.

Input:
A:v4.9
B:v3.19

Before: 0.09 s
 After: 0.09 s

To test the negative case, add a new commit with parent v3.19,
regenerate the commit-graph, and then run with B pointing at that
commit.

Before: 0.43 s
 After: 0.09 s

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotest-reach: test commit_contains
Derrick Stolee [Fri, 20 Jul 2018 16:33:25 +0000 (16:33 +0000)]
test-reach: test commit_contains

The commit_contains method has two modes which depend on the given
ref_filter struct. We have the "normal" algorithm (which is also the
typically-slow operation) and the "tag" algorithm. This difference is
essentially what changes performance for 'git branch --contains' versus
'git tag --contains'. There are thoughts that the data shapes used by
these two applications justify the different implementations.

Create tests using 'test-tool reach commit_contains [--tag]' to cover
both methods.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotest-reach: test can_all_from_reach_with_flags
Derrick Stolee [Fri, 20 Jul 2018 16:33:23 +0000 (16:33 +0000)]
test-reach: test can_all_from_reach_with_flags

The can_all_from_reach_with_flags method is used by ok_to_give_up in
upload-pack.c to see if we have done enough negotiation during a fetch.
This method is intentionally created to preserve state between calls to
assist with stateful negotiation, such as over SSH.

To make this method testable, add a new can_all_from_reach method that
does the initial setup and final tear-down. We will later use this
method in production code. Call the method from 'test-tool reach' for
now.

Since this is a many-to-many reachability query, add a new type of input
to the 'test-tool reach' input format. Lines "Y:<committish>" create a
list of commits to be the reachability targets from the commits in the
'X' list. In the context of fetch negotiation, the 'X' commits are the
'want' commits and the 'Y' commits are the 'have' commits.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotest-reach: test reduce_heads
Derrick Stolee [Fri, 20 Jul 2018 16:33:22 +0000 (16:33 +0000)]
test-reach: test reduce_heads

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotest-reach: test get_merge_bases_many
Derrick Stolee [Fri, 20 Jul 2018 16:33:20 +0000 (16:33 +0000)]
test-reach: test get_merge_bases_many

The get_merge_bases_many method returns a list of merge bases for a
single commit (A) against a list of commits (X). Some care is needed in
constructing the expected behavior because the result is not the
expected merge-base for an octopus merge with those parents but instead
the set of maximal commits that are reachable from A and at least one of
the commits in X.

Add get_merge_bases_many to 'test-tool reach' and create a test that
demonstrates that this output returns multiple results. Specifically, we
select a list of three commits such that we output two commits that are
reachable from one of the first two, respectively, and none are
reachable from the third.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotest-reach: test is_descendant_of
Derrick Stolee [Fri, 20 Jul 2018 16:33:18 +0000 (16:33 +0000)]
test-reach: test is_descendant_of

The is_descendant_of method takes a single commit as its first parameter
and a list of commits as its second parameter. Extend the input of the
'test-tool reach' command to take multiple lines of the form
"X:<committish>" to construct a list of commits. Pass these to
is_descendant_of and create tests that check each result.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotest-reach: test in_merge_bases
Derrick Stolee [Fri, 20 Jul 2018 16:33:17 +0000 (16:33 +0000)]
test-reach: test in_merge_bases

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotest-reach: create new test tool for ref_newer
Derrick Stolee [Fri, 20 Jul 2018 16:33:15 +0000 (16:33 +0000)]
test-reach: create new test tool for ref_newer

As we prepare to change the behavior of the algorithms in
commit-reach.c, create a new test-tool subcommand 'reach' to test these
methods on interesting commit-graph shapes.

To use the new test-tool, use 'test-tool reach <method>' and provide
input to stdin that describes the inputs to the method. Currently, we
only implement the ref_newer method, which requires two commits. Use
lines "A:<committish>" and "B:<committish>" for the two inputs. We will
expand this input later to accommodate methods that take lists of
commits.

The test t6600-test-reach.sh creates a repo whose commits form a
two-dimensional grid. This grid makes it easy for us to determine
reachability because commit-A-B can reach commit-X-Y if and only if A is
at least X and B is at least Y. This helps create interesting test cases
for each result of the methods in commit-reach.c.

We test all methods in three different states of the commit-graph file:
Non-existent (no generation numbers), fully computed, and mixed (some
commits have generation numbers and others do not).

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-reach: move can_all_from_reach_with_flags
Derrick Stolee [Fri, 20 Jul 2018 16:33:13 +0000 (16:33 +0000)]
commit-reach: move can_all_from_reach_with_flags

There are several commit walks in the codebase. Group them together into
a new commit-reach.c file and corresponding header. After we group these
walks into one place, we can reduce duplicate logic by calling
equivalent methods.

The can_all_from_reach_with_flags method is used in a stateful way by
upload-pack.c. The parameters are very flexible, so we will be able to
use its commit walking logic for many other callers.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoupload-pack: generalize commit date cutoff
Derrick Stolee [Fri, 20 Jul 2018 16:33:12 +0000 (16:33 +0000)]
upload-pack: generalize commit date cutoff

The ok_to_give_up() method uses the commit date as a cutoff to avoid
walking the entire reachble set of commits. Before moving the
reachable() method to commit-reach.c, pull out the dependence on the
global constant 'oldest_have' with a 'min_commit_date' parameter.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoupload-pack: refactor ok_to_give_up()
Derrick Stolee [Fri, 20 Jul 2018 16:33:11 +0000 (16:33 +0000)]
upload-pack: refactor ok_to_give_up()

In anticipation of consolidating all commit reachability algorithms,
refactor ok_to_give_up() in order to allow splitting its logic into
an external method.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoupload-pack: make reachable() more generic
Derrick Stolee [Fri, 20 Jul 2018 16:33:09 +0000 (16:33 +0000)]
upload-pack: make reachable() more generic

In anticipation of moving the reachable() method to commit-reach.c,
modify the prototype to be more generic to flags known outside of
upload-pack.c. Also rename 'want' to 'from' to make the statement
more clear outside of the context of haves/wants negotiation.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-reach: move commit_contains from ref-filter
Derrick Stolee [Fri, 20 Jul 2018 16:33:08 +0000 (16:33 +0000)]
commit-reach: move commit_contains from ref-filter

There are several commit walks in the codebase. Group them together into
a new commit-reach.c file and corresponding header. After we group these
walks into one place, we can reduce duplicate logic by calling
equivalent methods.

All methods are direct moves, except we also make the commit_contains()
method public so its consumers in ref-filter.c can still call it. We can
also test this method in a test-tool in a later commit.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-reach: move ref_newer from remote.c
Derrick Stolee [Fri, 20 Jul 2018 16:33:06 +0000 (16:33 +0000)]
commit-reach: move ref_newer from remote.c

There are several commit walks in the codebase. Group them together into
a new commit-reach.c file and corresponding header. After we group these
walks into one place, we can reduce duplicate logic by calling
equivalent methods.

The ref_newer() method is used by 'git push -f' to check if a force-push
is necessary. By making the method public, we make it possible to test
the method directly without setting up an envieronment where a 'git
push' call makes sense.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit.h: remove method declarations
Derrick Stolee [Fri, 20 Jul 2018 16:33:04 +0000 (16:33 +0000)]
commit.h: remove method declarations

These methods are now declared in commit-reach.h. Remove them from
commit.h and add new include statements in all files that require these
declarations.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-reach: move walk methods from commit.c
Derrick Stolee [Fri, 20 Jul 2018 16:33:02 +0000 (16:33 +0000)]
commit-reach: move walk methods from commit.c

There are several commit walks in the codebase. Group them together into
a new commit-reach.c file and corresponding header. After we group these
walks into one place, we can reduce duplicate logic by calling
equivalent methods.

The method declarations in commit.h are not touched by this commit and
will be moved in a following commit. Many consumers need to point to
commit-reach.h and that would bloat this commit.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: add repo arg to graph readers
Jonathan Tan [Wed, 11 Jul 2018 22:42:42 +0000 (15:42 -0700)]
commit-graph: add repo arg to graph readers

Add a struct repository argument to the functions in commit-graph.h that
read the commit graph. (This commit does not affect functions that write
commit graphs.)

Because the commit graph functions can now read the commit graph of any
repository, the global variable core_commit_graph has been removed.
Instead, the config option core.commitGraph is now read on the first
time in a repository that a commit is attempted to be parsed using its
commit graph.

This commit includes a test that exercises the functionality on an
arbitrary repository that is not the_repository.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: store graph in struct object_store
Jonathan Tan [Wed, 11 Jul 2018 22:42:41 +0000 (15:42 -0700)]
commit-graph: store graph in struct object_store

Instead of storing commit graphs in static variables, store them in
struct object_store. There are no changes to the signatures of existing
functions - they all still only support the_repository, and support for
other instances of struct repository will be added in a subsequent
commit.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: add free_commit_graph
Jonathan Tan [Wed, 11 Jul 2018 22:42:40 +0000 (15:42 -0700)]
commit-graph: add free_commit_graph

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: add missing forward declaration
Jonathan Tan [Wed, 11 Jul 2018 22:42:39 +0000 (15:42 -0700)]
commit-graph: add missing forward declaration

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoobject-store: add missing include
Jonathan Tan [Wed, 11 Jul 2018 22:42:38 +0000 (15:42 -0700)]
object-store: add missing include

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: refactor preparing commit graph
Jonathan Tan [Wed, 11 Jul 2018 22:42:37 +0000 (15:42 -0700)]
commit-graph: refactor preparing commit graph

Two functions in the code (1) check if the repository is configured for
commit graphs, (2) call prepare_commit_graph(), and (3) check if the
graph exists. Move (1) and (3) into prepare_commit_graph(), reducing
duplication of code.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoMerge branch 'ds/commit-graph-fsck' into jt/commit-graph-per-object-store
Junio C Hamano [Tue, 17 Jul 2018 22:46:19 +0000 (15:46 -0700)]
Merge branch 'ds/commit-graph-fsck' into jt/commit-graph-per-object-store

* ds/commit-graph-fsck: (23 commits)
  coccinelle: update commit.cocci
  commit-graph: update design document
  gc: automatically write commit-graph files
  commit-graph: add '--reachable' option
  commit-graph: use string-list API for input
  fsck: verify commit-graph
  commit-graph: verify contents match checksum
  commit-graph: test for corrupted octopus edge
  commit-graph: verify commit date
  commit-graph: verify generation number
  commit-graph: verify parent list
  commit-graph: verify root tree OIDs
  commit-graph: verify objects exist
  commit-graph: verify corrupt OID fanout and lookup
  commit-graph: verify required chunks are present
  commit-graph: verify catches corrupt signature
  commit-graph: add 'verify' subcommand
  commit-graph: load a root tree from specific graph
  commit: force commit to parse from object database
  commit-graph: parse commit from chosen graph
  ...

2 years agococcinelle: update commit.cocci
Derrick Stolee [Fri, 13 Jul 2018 16:30:46 +0000 (16:30 +0000)]
coccinelle: update commit.cocci

A recent patch series renamed the get_commit_tree_from_graph method but
forgot to update the coccinelle script that exempted it from rules
regarding accesses to 'maybe_tree'. This fixes that oversight to bring
the coccinelle scripts back to a good state.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit.c: allow lookup_commit_reference to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:22 +0000 (18:22 -0700)]
commit.c: allow lookup_commit_reference to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit.c: allow lookup_commit_reference_gently to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:21 +0000 (18:22 -0700)]
commit.c: allow lookup_commit_reference_gently to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotag.c: allow deref_tag to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:20 +0000 (18:22 -0700)]
tag.c: allow deref_tag to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoobject.c: allow parse_object to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:19 +0000 (18:22 -0700)]
object.c: allow parse_object to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoobject.c: allow parse_object_buffer to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:18 +0000 (18:22 -0700)]
object.c: allow parse_object_buffer to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit.c: allow get_cached_commit_buffer to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:17 +0000 (18:22 -0700)]
commit.c: allow get_cached_commit_buffer to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit.c: allow set_commit_buffer to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:16 +0000 (18:22 -0700)]
commit.c: allow set_commit_buffer to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit.c: migrate the commit buffer to the parsed object store
Stefan Beller [Fri, 29 Jun 2018 01:22:15 +0000 (18:22 -0700)]
commit.c: migrate the commit buffer to the parsed object store

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-slabs: remove realloc counter outside of slab struct
Stefan Beller [Fri, 29 Jun 2018 01:22:14 +0000 (18:22 -0700)]
commit-slabs: remove realloc counter outside of slab struct

The realloc counter is declared outside the struct for the given slabname,
which makes it harder for a follow up patch to move the declaration of the
struct around as then the counter variable would need special treatment.

As the reallocation counter is currently unused we can just remove it.
If we ever need to count the reallocations again, we can reintroduce
the counter as part of 'struct slabname' in commit-slab-decl.h.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit.c: allow parse_commit_buffer to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:13 +0000 (18:22 -0700)]
commit.c: allow parse_commit_buffer to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotag: allow parse_tag_buffer to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:12 +0000 (18:22 -0700)]
tag: allow parse_tag_buffer to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotag: allow lookup_tag to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:11 +0000 (18:22 -0700)]
tag: allow lookup_tag to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit: allow lookup_commit to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:10 +0000 (18:22 -0700)]
commit: allow lookup_commit to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotree: allow lookup_tree to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:09 +0000 (18:22 -0700)]
tree: allow lookup_tree to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoblob: allow lookup_blob to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:08 +0000 (18:22 -0700)]
blob: allow lookup_blob to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoobject: allow lookup_object to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:07 +0000 (18:22 -0700)]
object: allow lookup_object to handle arbitrary repositories

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoobject: allow object_as_type to handle arbitrary repositories
Stefan Beller [Fri, 29 Jun 2018 01:22:06 +0000 (18:22 -0700)]
object: allow object_as_type to handle arbitrary repositories

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotag: add repository argument to deref_tag
Stefan Beller [Fri, 29 Jun 2018 01:22:05 +0000 (18:22 -0700)]
tag: add repository argument to deref_tag

Add a repository argument to allow the callers of deref_tag
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotag: add repository argument to parse_tag_buffer
Stefan Beller [Fri, 29 Jun 2018 01:22:04 +0000 (18:22 -0700)]
tag: add repository argument to parse_tag_buffer

Add a repository argument to allow the callers of parse_tag_buffer
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotag: add repository argument to lookup_tag
Stefan Beller [Fri, 29 Jun 2018 01:22:03 +0000 (18:22 -0700)]
tag: add repository argument to lookup_tag

Add a repository argument to allow the callers of lookup_tag
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit: add repository argument to get_cached_commit_buffer
Stefan Beller [Fri, 29 Jun 2018 01:22:02 +0000 (18:22 -0700)]
commit: add repository argument to get_cached_commit_buffer

Add a repository argument to allow callers of get_cached_commit_buffer to
be more specific about which repository to handle. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit: add repository argument to set_commit_buffer
Stefan Beller [Fri, 29 Jun 2018 01:22:01 +0000 (18:22 -0700)]
commit: add repository argument to set_commit_buffer

Add a repository argument to allow callers of set_commit_buffer to
be more specific about which repository to handle. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit: add repository argument to parse_commit_buffer
Stefan Beller [Fri, 29 Jun 2018 01:22:00 +0000 (18:22 -0700)]
commit: add repository argument to parse_commit_buffer

Add a repository argument to allow the callers of parse_commit_buffer
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit: add repository argument to lookup_commit
Stefan Beller [Fri, 29 Jun 2018 01:21:59 +0000 (18:21 -0700)]
commit: add repository argument to lookup_commit

Add a repository argument to allow callers of lookup_commit to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit: add repository argument to lookup_commit_reference
Stefan Beller [Fri, 29 Jun 2018 01:21:58 +0000 (18:21 -0700)]
commit: add repository argument to lookup_commit_reference

Add a repository argument to allow callers of lookup_commit_reference
to be more specific about which repository to handle. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit: add repository argument to lookup_commit_reference_gently
Stefan Beller [Fri, 29 Jun 2018 01:21:57 +0000 (18:21 -0700)]
commit: add repository argument to lookup_commit_reference_gently

Add a repository argument to allow callers of
lookup_commit_reference_gently to be more specific about which
repository to handle. This is a small mechanical change; it doesn't
change the implementation to handle repositories other than
the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agotree: add repository argument to lookup_tree
Stefan Beller [Fri, 29 Jun 2018 01:21:56 +0000 (18:21 -0700)]
tree: add repository argument to lookup_tree

Add a repository argument to allow the callers of lookup_tree
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoblob: add repository argument to lookup_blob
Stefan Beller [Fri, 29 Jun 2018 01:21:55 +0000 (18:21 -0700)]
blob: add repository argument to lookup_blob

Add a repository argument to allow the callers of lookup_blob
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoobject: add repository argument to object_as_type
Stefan Beller [Fri, 29 Jun 2018 01:21:54 +0000 (18:21 -0700)]
object: add repository argument to object_as_type

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoobject: add repository argument to parse_object_buffer
Stefan Beller [Fri, 29 Jun 2018 01:21:53 +0000 (18:21 -0700)]
object: add repository argument to parse_object_buffer

Add a repository argument to allow the callers of parse_object_buffer
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoobject: add repository argument to lookup_object
Stefan Beller [Fri, 29 Jun 2018 01:21:52 +0000 (18:21 -0700)]
object: add repository argument to lookup_object

Add a repository argument to allow callers of lookup_object to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoobject: add repository argument to parse_object
Stefan Beller [Fri, 29 Jun 2018 01:21:51 +0000 (18:21 -0700)]
object: add repository argument to parse_object

Add a repository argument to allow the callers of parse_object
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoMerge branch 'sb/object-store-grafts' into sb/object-store-lookup
Junio C Hamano [Fri, 29 Jun 2018 17:24:33 +0000 (10:24 -0700)]
Merge branch 'sb/object-store-grafts' into sb/object-store-lookup

* sb/object-store-grafts:
  commit: allow lookup_commit_graft to handle arbitrary repositories
  commit: allow prepare_commit_graft to handle arbitrary repositories
  shallow: migrate shallow information into the object parser
  path.c: migrate global git_path_* to take a repository argument
  cache: convert get_graft_file to handle arbitrary repositories
  commit: convert read_graft_file to handle arbitrary repositories
  commit: convert register_commit_graft to handle arbitrary repositories
  commit: convert commit_graft_pos() to handle arbitrary repositories
  shallow: add repository argument to is_repository_shallow
  shallow: add repository argument to check_shallow_file_for_update
  shallow: add repository argument to register_shallow
  shallow: add repository argument to set_alternate_shallow_file
  commit: add repository argument to lookup_commit_graft
  commit: add repository argument to prepare_commit_graft
  commit: add repository argument to read_graft_file
  commit: add repository argument to register_commit_graft
  commit: add repository argument to commit_graft_pos
  object: move grafts to object parser
  object-store: move object access functions to object-store.h

2 years agoSecond batch for 2.19 cycle
Junio C Hamano [Thu, 28 Jun 2018 19:55:47 +0000 (12:55 -0700)]
Second batch for 2.19 cycle

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoMerge branch 'sb/fix-fetching-moved-submodules'
Junio C Hamano [Thu, 28 Jun 2018 19:53:34 +0000 (12:53 -0700)]
Merge branch 'sb/fix-fetching-moved-submodules'

The code to try seeing if a fetch is necessary in a submodule
during a fetch with --recurse-submodules got confused when the path
to the submodule was changed in the range of commits in the
superproject, sometimes showing "(null)".  This has been corrected.

* sb/fix-fetching-moved-submodules:
  t5526: test recursive submodules when fetching moved submodules
  submodule: fix NULL correctness in renamed broken submodules

2 years agoMerge branch 'tz/cred-netrc-cleanup'
Junio C Hamano [Thu, 28 Jun 2018 19:53:33 +0000 (12:53 -0700)]
Merge branch 'tz/cred-netrc-cleanup'

Build and test procedure for netrc credential helper (in contrib/)
has been updated.

* tz/cred-netrc-cleanup:
  git-credential-netrc: make "all" default target of Makefile
  git-credential-netrc: fix exit status when tests fail
  git-credential-netrc: use in-tree Git.pm for tests
  git-credential-netrc: minor whitespace cleanup in test script

2 years agoMerge branch 'jc/clean-after-sanity-tests'
Junio C Hamano [Thu, 28 Jun 2018 19:53:33 +0000 (12:53 -0700)]
Merge branch 'jc/clean-after-sanity-tests'

test cleanup.

* jc/clean-after-sanity-tests:
  tests: clean after SANITY tests

2 years agoMerge branch 'nd/completion-negation'
Junio C Hamano [Thu, 28 Jun 2018 19:53:32 +0000 (12:53 -0700)]
Merge branch 'nd/completion-negation'

Continuing with the idea to programmatically enumerate various
pieces of data required for command line completion, the codebase
has been taught to enumerate options prefixed with "--no-" to
negate them.

* nd/completion-negation:
  completion: collapse extra --no-.. options
  completion: suppress some -no- options
  parse-options: option to let --git-completion-helper show negative form

2 years agoMerge branch 'pw/add-p-recount'
Junio C Hamano [Thu, 28 Jun 2018 19:53:32 +0000 (12:53 -0700)]
Merge branch 'pw/add-p-recount'

When user edits the patch in "git add -p" and the user's editor is
set to strip trailing whitespaces indiscriminately, an empty line
that is unchanged in the patch would become completely empty
(instead of a line with a sole SP on it).  The code introduced in
Git 2.17 timeframe failed to parse such a patch, but now it learned
to notice the situation and cope with it.

* pw/add-p-recount:
  add -p: fix counting empty context lines in edited patches

2 years agoMerge branch 'jk/fetch-all-peeled-fix'
Junio C Hamano [Thu, 28 Jun 2018 19:53:32 +0000 (12:53 -0700)]
Merge branch 'jk/fetch-all-peeled-fix'

"git fetch-pack --all" used to unnecessarily fail upon seeing an
annotated tag that points at an object other than a commit.

* jk/fetch-all-peeled-fix:
  fetch-pack: test explicitly that --all can fetch tag references pointing to non-commits
  fetch-pack: don't try to fetch peel values with --all

2 years agoMerge branch 'ms/send-pack-honor-config'
Junio C Hamano [Thu, 28 Jun 2018 19:53:30 +0000 (12:53 -0700)]
Merge branch 'ms/send-pack-honor-config'

"git send-pack --signed" (hence "git push --signed" over the http
transport) did not read user ident from the config mechanism to
determine whom to sign the push certificate as, which has been
corrected.

* ms/send-pack-honor-config:
  builtin/send-pack: populate the default configs

2 years agoMerge branch 'jh/partial-clone'
Junio C Hamano [Thu, 28 Jun 2018 19:53:30 +0000 (12:53 -0700)]
Merge branch 'jh/partial-clone'

The recent addition of "partial clone" experimental feature kicked
in when it shouldn't, namely, when there is no partial-clone filter
defined even if extensions.partialclone is set.

* jh/partial-clone:
  list-objects: check if filter is NULL before using

2 years agoMerge branch 'sg/gpg-tests-fix'
Junio C Hamano [Thu, 28 Jun 2018 19:53:29 +0000 (12:53 -0700)]
Merge branch 'sg/gpg-tests-fix'

Some flaky tests have been fixed.

* sg/gpg-tests-fix:
  tests: make forging GPG signed commits and tags more robust
  t7510-signed-commit: use 'test_must_fail'

2 years agoMerge branch 'as/safecrlf-quiet-fix'
Junio C Hamano [Thu, 28 Jun 2018 19:53:29 +0000 (12:53 -0700)]
Merge branch 'as/safecrlf-quiet-fix'

Fix for 2.17-era regression around `core.safecrlf`.

* as/safecrlf-quiet-fix:
  config.c: fix regression for core.safecrlf false

2 years agoMerge branch 'ab/refspec-init-fix'
Junio C Hamano [Thu, 28 Jun 2018 19:53:29 +0000 (12:53 -0700)]
Merge branch 'ab/refspec-init-fix'

Make refspec parsing codepath more robust.

* ab/refspec-init-fix:
  refspec: initalize `refspec_item` in `valid_fetch_refspec()`
  refspec: add back a refspec_item_init() function
  refspec: s/refspec_item_init/&_or_die/g

2 years agocommit-graph: update design document
Derrick Stolee [Wed, 27 Jun 2018 13:24:47 +0000 (09:24 -0400)]
commit-graph: update design document

The commit-graph feature is now integrated with 'fsck' and 'gc',
so remove those items from the "Future Work" section of the
commit-graph design document.

Also remove the section on lazy-loading trees, as that was completed
in an earlier patch series.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agogc: automatically write commit-graph files
Derrick Stolee [Wed, 27 Jun 2018 13:24:46 +0000 (09:24 -0400)]
gc: automatically write commit-graph files

The commit-graph file is a very helpful feature for speeding up git
operations. In order to make it more useful, make it possible to
write the commit-graph file during standard garbage collection
operations.

Add a 'gc.commitGraph' config setting that triggers writing a
commit-graph file after any non-trivial 'git gc' command. Defaults to
false while the commit-graph feature matures. We specifically do not
want to have this on by default until the commit-graph feature is fully
integrated with history-modifying features like shallow clones.

Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: add '--reachable' option
Derrick Stolee [Wed, 27 Jun 2018 13:24:45 +0000 (09:24 -0400)]
commit-graph: add '--reachable' option

When writing commit-graph files, it can be convenient to ask for all
reachable commits (starting at the ref set) in the resulting file. This
is particularly helpful when writing to stdin is complicated, such as a
future integration with 'git gc'.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: use string-list API for input
Derrick Stolee [Wed, 27 Jun 2018 13:24:44 +0000 (09:24 -0400)]
commit-graph: use string-list API for input

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agofsck: verify commit-graph
Derrick Stolee [Wed, 27 Jun 2018 13:24:43 +0000 (09:24 -0400)]
fsck: verify commit-graph

If core.commitGraph is true, verify the contents of the commit-graph
during 'git fsck' using the 'git commit-graph verify' subcommand. Run
this check on all alternates, as well.

We use a new process for two reasons:

1. The subcommand decouples the details of loading and verifying a
   commit-graph file from the other fsck details.

2. The commit-graph verification requires the commits to be loaded
   in a specific order to guarantee we parse from the commit-graph
   file for some objects and from the object database for others.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: verify contents match checksum
Derrick Stolee [Wed, 27 Jun 2018 13:24:42 +0000 (09:24 -0400)]
commit-graph: verify contents match checksum

The commit-graph file ends with a SHA1 hash of the previous contents. If
a commit-graph file has errors but the checksum hash is correct, then we
know that the problem is a bug in Git and not simply file corruption
after-the-fact.

Compute the checksum right away so it is the first error that appears,
and make the message translatable since this error can be "corrected" by
a user by simply deleting the file and recomputing. The rest of the
errors are useful only to developers.

Be sure to continue checking the rest of the file data if the checksum
is wrong. This is important for our tests, as we break the checksum as
we modify bytes of the commit-graph file.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: test for corrupted octopus edge
Derrick Stolee [Wed, 27 Jun 2018 13:24:41 +0000 (09:24 -0400)]
commit-graph: test for corrupted octopus edge

The commit-graph file has an extra chunk to store the parent int-ids for
parents beyond the first parent for octopus merges. Our test repo has a
single octopus merge that we can manipulate to demonstrate the 'verify'
subcommand detects incorrect values in that chunk.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: verify commit date
Derrick Stolee [Wed, 27 Jun 2018 13:24:40 +0000 (09:24 -0400)]
commit-graph: verify commit date

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: verify generation number
Derrick Stolee [Wed, 27 Jun 2018 13:24:39 +0000 (09:24 -0400)]
commit-graph: verify generation number

While iterating through the commit parents, perform the generation
number calculation and compare against the value stored in the
commit-graph.

The tests demonstrate that having a different set of parents affects
the generation number calculation, and this value propagates to
descendants. Hence, we drop the single-line condition on the output.

Since Git will ship with the commit-graph feature without generation
numbers, we need to accept commit-graphs with all generation numbers
equal to zero. In this case, ignore the generation number calculation.

However, verify that we should never have a mix of zero and non-zero
generation numbers. Create a test that sets one commit to generation
zero and all following commits report a failure as they have non-zero
generation in a file that contains generation number zero.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: verify parent list
Derrick Stolee [Wed, 27 Jun 2018 13:24:38 +0000 (09:24 -0400)]
commit-graph: verify parent list

The commit-graph file stores parents in a two-column portion of the
commit data chunk. If there is only one parent, then the second column
stores 0xFFFFFFFF to indicate no second parent.

The 'verify' subcommand checks the parent list for the commit loaded
from the commit-graph and the one parsed from the object database. Test
these checks for corrupt parents, too many parents, and wrong parents.

Add a boundary check to insert_parent_or_die() for when the parent
position value is out of range.

The octopus merge will be tested in a later commit.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: verify root tree OIDs
Derrick Stolee [Wed, 27 Jun 2018 13:24:37 +0000 (09:24 -0400)]
commit-graph: verify root tree OIDs

The 'verify' subcommand must compare the commit content parsed from the
commit-graph against the content in the object database. Use
lookup_commit() and parse_commit_in_graph_one() to parse the commits
from the graph and compare against a commit that is loaded separately
and parsed directly from the object database.

Add checks for the root tree OID.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: verify objects exist
Derrick Stolee [Wed, 27 Jun 2018 13:24:36 +0000 (09:24 -0400)]
commit-graph: verify objects exist

In the 'verify' subcommand, load commits directly from the object
database to ensure they exist. Parse by skipping the commit-graph.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: verify corrupt OID fanout and lookup
Derrick Stolee [Wed, 27 Jun 2018 13:24:35 +0000 (09:24 -0400)]
commit-graph: verify corrupt OID fanout and lookup

In the commit-graph file, the OID fanout chunk provides an index into
the OID lookup. The 'verify' subcommand should find incorrect values
in the fanout.

Similarly, the 'verify' subcommand should find out-of-order values in
the OID lookup.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: verify required chunks are present
Derrick Stolee [Wed, 27 Jun 2018 13:24:34 +0000 (09:24 -0400)]
commit-graph: verify required chunks are present

The commit-graph file requires the following three chunks:

* OID Fanout
* OID Lookup
* Commit Data

If any of these are missing, then the 'verify' subcommand should
report a failure. This includes the chunk IDs malformed or the
chunk count is truncated.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: verify catches corrupt signature
Derrick Stolee [Wed, 27 Jun 2018 13:24:33 +0000 (09:24 -0400)]
commit-graph: verify catches corrupt signature

This is the first of several commits that add a test to check that
'git commit-graph verify' catches corruption in the commit-graph
file. The first test checks that the command catches an error in
the file signature. This is a check that exists in the existing
commit-graph reading code.

Add a helper method 'corrupt_graph_and_verify' to the test script
t5318-commit-graph.sh. This helper corrupts the commit-graph file
at a certain location, runs 'git commit-graph verify', and reports
the output to the 'err' file. This data is filtered to remove the
lines added by 'test_must_fail' when the test is run verbosely.
Then, the output is checked to contain a specific error message.

Most messages from 'git commit-graph verify' will not be marked
for translation. There will be one exception: the message that
reports an invalid checksum will be marked for translation, as that
is the only message that is intended for a typical user.

Helped-by: Szeder Gábor <szeder.dev@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: add 'verify' subcommand
Derrick Stolee [Wed, 27 Jun 2018 13:24:32 +0000 (09:24 -0400)]
commit-graph: add 'verify' subcommand

If the commit-graph file becomes corrupt, we need a way to verify
that its contents match the object database. In the manner of
'git fsck' we will implement a 'git commit-graph verify' subcommand
to report all issues with the file.

Add the 'verify' subcommand to the 'commit-graph' builtin and its
documentation. The subcommand is currently a no-op except for
loading the commit-graph into memory, which may trigger run-time
errors that would be caught by normal use. Add a simple test that
ensures the command returns a zero error code.

If no commit-graph file exists, this is an acceptable state. Do
not report any errors.

Helped-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: load a root tree from specific graph
Derrick Stolee [Wed, 27 Jun 2018 13:24:31 +0000 (09:24 -0400)]
commit-graph: load a root tree from specific graph

When lazy-loading a tree for a commit, it will be important to select
the tree from a specific struct commit_graph. Create a new method that
specifies the commit-graph file and use that in
get_commit_tree_in_graph().

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit: force commit to parse from object database
Derrick Stolee [Wed, 27 Jun 2018 13:24:30 +0000 (09:24 -0400)]
commit: force commit to parse from object database

In anticipation of verifying commit-graph file contents against the
object database, create parse_commit_internal() to allow side-stepping
the commit-graph file and parse directly from the object database.

Due to the use of generation numbers, this method should not be called
unless the intention is explicit in avoiding commits from the
commit-graph file.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: parse commit from chosen graph
Derrick Stolee [Wed, 27 Jun 2018 13:24:29 +0000 (09:24 -0400)]
commit-graph: parse commit from chosen graph

Before verifying a commit-graph file against the object database, we
need to parse all commits from the given commit-graph file. Create
parse_commit_in_graph_one() to target a given struct commit_graph.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: fix GRAPH_MIN_SIZE
Derrick Stolee [Wed, 27 Jun 2018 13:24:28 +0000 (09:24 -0400)]
commit-graph: fix GRAPH_MIN_SIZE

The GRAPH_MIN_SIZE macro should be the smallest size of a parsable
commit-graph file. However, the minimum number of chunks was wrong.
It is possible to write a commit-graph file with zero commits, and
that violates this macro's value.

Rewrite the macro, and use extra macros to better explain the magic
constants.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agocommit-graph: UNLEAK before die()
Derrick Stolee [Wed, 27 Jun 2018 13:24:27 +0000 (09:24 -0400)]
commit-graph: UNLEAK before die()

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agot5318-commit-graph.sh: use core.commitGraph
Derrick Stolee [Wed, 27 Jun 2018 13:24:26 +0000 (09:24 -0400)]
t5318-commit-graph.sh: use core.commitGraph

The commit-graph tests should be checking that normal Git operations
succeed and have matching output with and without the commit-graph
feature enabled. However, the test was toggling 'core.graph' instead
of the correct 'core.commitGraph' variable.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoFirst batch for 2.19 cycle
Junio C Hamano [Mon, 25 Jun 2018 20:27:15 +0000 (13:27 -0700)]
First batch for 2.19 cycle

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 years agoMerge branch 'sb/plug-misc-leaks'
Junio C Hamano [Mon, 25 Jun 2018 20:22:41 +0000 (13:22 -0700)]
Merge branch 'sb/plug-misc-leaks'

Misc leak plugging.

* sb/plug-misc-leaks:
  sequencer.c: plug mem leak in git_sequencer_config
  sequencer.c: plug leaks in do_pick_commit
  submodule--helper: plug mem leak in print_default_remote
  refs/packed-backend.c: close fd of empty file

2 years agoMerge branch 'cc/tests-without-assuming-ref-files-backend'
Junio C Hamano [Mon, 25 Jun 2018 20:22:41 +0000 (13:22 -0700)]
Merge branch 'cc/tests-without-assuming-ref-files-backend'

Instead of mucking with filesystem directly, use plumbing commands
update-ref etc. to manipulate the refs in the tests.

* cc/tests-without-assuming-ref-files-backend:
  t9104: kosherly remove remote refs

2 years agoMerge branch 'sg/update-ref-stdin-cleanup'
Junio C Hamano [Mon, 25 Jun 2018 20:22:40 +0000 (13:22 -0700)]
Merge branch 'sg/update-ref-stdin-cleanup'

Code cleanup.

* sg/update-ref-stdin-cleanup:
  update-ref --stdin: use skip_prefix()