This changes the TLS and certificate handling in a few ways:
- We always use TLS 1.2, both for sync connections (as previously) and
the GUI/REST/discovery stuff. This is a tightening of the requirements
on the GUI. AS far as I can tell from caniusethis.com every browser from
2013 and forward supports TLS 1.2, so I think we should be fine.
- We always greate ECDSA certificates. Previously we'd create
ECDSA-with-RSA certificates for sync connections and pure RSA
certificates for the web stuff. The new default is more modern and the
same everywhere. These certificates are OK in TLS 1.2.
- We use the Go CPU detection stuff to choose the cipher suites to use,
indirectly. The TLS package uses CPU capabilities probing to select
either AES-GCM (fast if we have AES-NI) or ChaCha20 (faster if we
don't). These CPU detection things aren't exported though, so the tlsutil
package now does a quick TLS handshake with itself as part of init().
If the chosen cipher suite was AES-GCM we prioritize that, otherwise we
prefer ChaCha20. Some might call this ugly. I think it's awesome.
In a recent change (#5201) this return disappeared. The effect is that
we first shortcut the file and then also treat it normally. This results
in to database updates after each other, which are bound to end up in
the same batch. This means we remove one sequence entry and add two.
Not marking the issues as fixed, because I need to do more testing and
there are other discrepancies...
This adds a thin type that holds the state associated with the
leveldb.DB, leaving the huge Instance type more or less stateless. Also
moves some keying stuff into the DB package so that other packages need
not know the keying specifics.
(This does not, yet, fix the cmd/stindex program, in order to keep the
diff size down. Hence the keying constants are still exported.)
* lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes#5233)
This is the somewhat ugly - but on the other hand clear - fix for what
is really a somewhat thorny issue. To avoid zombie folder runners a new
mutex is introduced that protects the RestartFolder operation. I hate
adding more mutexes but the alternatives I can think of are worse.
The other part of it is that the POST /rest/system/config operation now
waits for the config commit to complete. The point of this is that until
the commit has completed we should not accept another config commit. If
we did, we could end up with two separate RestartFolders queued in the
background. While they are both correct, and will run without
interfering with each other, we can't guarantee the order in which they
will run. Thus it could happen that the newer config got committed
first, and the older config commited after that, leaving us with the
wrong config running.
* test
* wip
* hax
* hax
* unflake test
* per folder mutexes
* paranoia
* race
* lib/fs: Add fakefs
This adds a new fake filesystem type. It's described rather extensively
in fakefs.go, but the main point is that it's for testing: when you want
to spin up a Syncthing and have a terabyte or two of random files that
can be synced somewhere, or an inifitely large filesystem to sync files
into.
It has pseudorandom properties such that data read from one fakefs can
be written into another fakefs and read back and it will look
consistent, without any of the data actually being stored.
To use:
<folder id="default" path="whatever" ...>
<filesystemType>fake</filesystemType>
This will create an empty fake filesystem. You can also specify that it
should be prefilled with files:
<folder id="default" path="whatever?size=2000000" ...>
<filesystemType>fake</filesystemType>
This will create a filesystem filled with 2TB of random data that can be
scanned and synced. There are more options, see fakefs.go.
Prefilled data is based on a deterministic seed, so you can index the
data and restart Syncthing and the index is still correct for all the
stored data.
The previous "Bad Request" was really confusing as it implies it's
somethign wrong with the request, which there isn't - the problem is
that server configuration forbids the request.
This removes the user and group juggling, which would fail when given
for example a PGID that already existed as the "syncthing" group could
then not be created with that PGID. It's not reasonable to expect the
user to know which group/user names/IDs are already present in the
Docker image.
Instead we now just launch under the specified IDs, while manually
setting the HOME env var to give us a home directory - the only thing we
needed the user entry for anyway.
Also updates to Go 1.11 and building without upgrades instead of
disabling by env var.
* release:
lib/model: Fixes on receive-only test setup and pulling (#5136)
lib/fs: Don't add path separators at end of path (fixes#5144) (#5146)
lib/fs: Evaluate root when watching not on fs creation (fixes#5043) (#5105)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.
This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).
The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.
This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).
The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
Removes the manual handling of the AUTHORS file, giving every committer automatic credit for their contribution.
Manual tweaks to the file are still accepted and retained by the scripts.
Adds a receive only folder type that does not send changes, and where the user can optionally revert local changes. Also changes some of the icons to make the three folder types distinguishable.
We have the invalid bit to indicate that a file isn't good. That's enough for remote devices. For ourselves, it would be good to know sometimes why the file isn't good - because it's an unsupported type, because it matches an ignore pattern, or because we detected the data is bad and we need to rescan it.
Or, and this is the main future reason for the PR, because it's a change detected on a receive only device. We will want something like the invalid flag for those changes, but marking them as invalid today means the scanner will rehash them. Hence something more fine grained is required.
This introduces a LocalFlags fields to the FileInfo where we can stash things that we care about locally. For example,
FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
FlagLocalIgnored = 1 << 1 // Matches local ignore patterns
FlagLocalMustRescan = 1 << 2 // Doesn't match content on disk, must be rechecked fully
The LocalFlags fields isn't sent over the wire; instead the Invalid attribute is calculated based on the flags at index sending time. It's on the FileInfo anyway because that's what we serialize to database etc.
The actual Invalid flag should after this just be considered when building the global state and figuring out availability for remote devices. It is not used for local file index entries.
It's pretty much the GitHub standard, says the obvious, and does it
better than our old one which was a little bit rambling.
Intentionally just doing it instead of discussing it, in the hope of
avoiding trolling. So sue me.
I'm trying to slowly clean this up a bit, and moving functionality out
into the folder types and having those methods not reach into model is
part of it. That can mean takign some odd arguments in the meantime,
some of those should probably become interfaces or properties on folder
in the long term.
The functionality was anyway mostly implemented there and not isolated
in the folderScanner type. The attempt to refactor it out in the other
direction wouldn't work given that the event loop and stuff is on
`folder`.
This adds a couple of utilities for transporting databases in JSON and a
test to load a database and verify a couple of invalid bits. The test
itself is quite pointless at the moment, but it lays the groundwork for
testing the migration of this data in the next step (after the invalid
bit should be changed to local flags for local files).
When that happens we need to have a database in the old format already
there in order to be able to test the migration.
