Add LocalFlags to FileInfo (#4952)

We have the invalid bit to indicate that a file isn't good. That's enough for remote devices. For ourselves, it would be good to know sometimes why the file isn't good - because it's an unsupported type, because it matches an ignore pattern, or because we detected the data is bad and we need to rescan it.

Or, and this is the main future reason for the PR, because it's a change detected on a receive only device. We will want something like the invalid flag for those changes, but marking them as invalid today means the scanner will rehash them. Hence something more fine grained is required.

This introduces a LocalFlags fields to the FileInfo where we can stash things that we care about locally. For example,

    FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
    FlagLocalIgnored     = 1 << 1 // Matches local ignore patterns
    FlagLocalMustRescan  = 1 << 2 // Doesn't match content on disk, must be rechecked fully

The LocalFlags fields isn't sent over the wire; instead the Invalid attribute is calculated based on the flags at index sending time. It's on the FileInfo anyway because that's what we serialize to database etc.

The actual Invalid flag should after this just be considered when building the global state and figuring out availability for remote devices. It is not used for local file index entries.

lib/scanner/walk.go

@@ -373,11 +373,12 @@ func (w *walker) walkRegular(ctx context.Context, relPath string, info fs.FileIn
 		if curFile.IsEquivalent(f, w.IgnorePerms, true) {
 			return nil
 		}
-		if curFile.Invalid {
-			// We do not want to override the global version with the file we
-			// currently have. Keeping only our local counter makes sure we are in
-			// conflict with any other existing versions, which will be resolved by
-			// the normal pulling mechanisms.
+		if curFile.ShouldConflict() {
+			// The old file was invalid for whatever reason and probably not
+			// up to date with what was out there in the cluster. Drop all
+			// others from the version vector to indicate that we haven't
+			// taken their version into account, and possibly cause a
+			// conflict.
 			f.Version = f.Version.DropOthers(w.ShortID)
 		}
 		l.Debugln("rescan:", curFile, info.ModTime().Unix(), info.Mode()&fs.ModePerm)
@@ -412,11 +413,12 @@ func (w *walker) walkDir(ctx context.Context, relPath string, info fs.FileInfo,
 		if cf.IsEquivalent(f, w.IgnorePerms, true) {
 			return nil
 		}
-		if cf.Invalid {
-			// We do not want to override the global version with the file we
-			// currently have. Keeping only our local counter makes sure we are in
-			// conflict with any other existing versions, which will be resolved by
-			// the normal pulling mechanisms.
+		if cf.ShouldConflict() {
+			// The old file was invalid for whatever reason and probably not
+			// up to date with what was out there in the cluster. Drop all
+			// others from the version vector to indicate that we haven't
+			// taken their version into account, and possibly cause a
+			// conflict.
 			f.Version = f.Version.DropOthers(w.ShortID)
 		}
 	}
@@ -467,11 +469,12 @@ func (w *walker) walkSymlink(ctx context.Context, relPath string, dchan chan pro
 		if cf.IsEquivalent(f, w.IgnorePerms, true) {
 			return nil
 		}
-		if cf.Invalid {
-			// We do not want to override the global version with the file we
-			// currently have. Keeping only our local counter makes sure we are in
-			// conflict with any other existing versions, which will be resolved by
-			// the normal pulling mechanisms.
+		if cf.ShouldConflict() {
+			// The old file was invalid for whatever reason and probably not
+			// up to date with what was out there in the cluster. Drop all
+			// others from the version vector to indicate that we haven't
+			// taken their version into account, and possibly cause a
+			// conflict.
 			f.Version = f.Version.DropOthers(w.ShortID)
 		}
 	}

lib/scanner/walk_test.go

@@ -648,10 +648,10 @@ func TestIssue4841(t *testing.T) {
 		Hashers:       2,
 		CurrentFiler: fakeCurrentFiler{
 			"foo": {
-				Name:    "foo",
-				Type:    protocol.FileInfoTypeFile,
-				Invalid: true,
-				Version: protocol.Vector{}.Update(1),
+				Name:       "foo",
+				Type:       protocol.FileInfoTypeFile,
+				LocalFlags: protocol.FlagLocalIgnored,
+				Version:    protocol.Vector{}.Update(1),
 			},
 		},
 		ShortID: protocol.LocalDeviceID.Short(),