perso/syncthing-arm
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Rework XDR encoding

Browse Source
This commit is contained in:
Jakob Borg
2014-02-20 17:40:15 +01:00
parent 87d473dc8f
commit 5837277f8d
27 changed files with 1843 additions and 1029 deletions
Download Patch File Download Diff File Expand all files Collapse all files

352
protocol/PROTOCOL.md
View File

@@ -1,26 +1,29 @@
Block Exchange Protocol v1.0
============================
Block Exchange Protocol v1
==========================
Introduction and Definitions
----------------------------
The BEP is used between two or more _nodes_ thus forming a _cluster_.
Each node has a _repository_ of files described by the _local model_,
containing modifications times and block hashes. The local model is sent
to the other nodes in the cluster. The union of all files in the local
models, with files selected for most recent modification time, forms the
_global model_. Each node strives to get it's repository in sync with
the global model by requesting missing blocks from the other nodes.
BEP is used between two or more _nodes_ thus forming a _cluster_. Each
node has one or more _repositories_ of files described by the _local
model_, containing metadata and block hashes. The local model is sent to
the other nodes in the cluster. The union of all files in the local
models, with files selected for highest change version, forms the
_global model_. Each node strives to get it's repositories in sync with
the global model by requesting missing or outdated blocks from the other
nodes in the cluster.
File data is described and transferred in units of _blocks_, each being
128 KiB (131072 bytes) in size.
Transport and Authentication
----------------------------
The BEP itself does not provide retransmissions, compression, encryption
nor authentication. It is expected that this is performed at lower
layers of the networking stack. A typical deployment stack should be
similar to the following:
BEP itself does not provide retransmissions, compression, encryption nor
authentication. It is expected that this is performed at lower layers of
the networking stack. The typical deployment stack is the following:
|-----------------------------|
+-----------------------------|
| Block Exchange Protocol |
|-----------------------------|
| Compression (RFC 1951) |
@@ -48,73 +51,127 @@ message boundary.
Messages
--------
Every message starts with one 32 bit word indicating the message version
and type. For BEP v1.0 the Version field is set to zero. Future versions
with incompatible message formats will increment the Version field. The
reserved bits must be set to zero.
Every message starts with one 32 bit word indicating the message
version, type and ID.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ver=0 | Message ID | Type | Reserved |
| Ver | Type | Message ID | Reply To |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
All data following the message header is in XDR (RFC 1014) encoding.
The actual data types in use by BEP, in XDR naming convention, are:
For BEP v1 the Version field is set to zero. Future versions with
incompatible message formats will increment the Version field.
The Type field indicates the type of data following the message header
and is one of the integers defined below.
The Message ID is set to a unique value for each transmitted message. In
request messages the Reply To is set to zero. In response messages it is
set to the message ID of the corresponding request.
All data following the message header is in XDR (RFC 1014) encoding. All
fields smaller than 32 bits and all variable length data is padded to a
multiple of 32 bits. The actual data types in use by BEP, in XDR naming
convention, are:
- (unsigned) int -- (unsigned) 32 bit integer
- (unsigned) hyper -- (unsigned) 64 bit integer
- opaque<> -- variable length opaque data
- string<> -- variable length string
The encoding of opaque<> and string<> are identical, the distinction is
solely in interpretation. Opaque data should not be interpreted as such,
but can be compared bytewise to other opaque data. All strings use the
UTF-8 encoding.
The transmitted length of string and opaque data is the length of actual
data, excluding any added padding. The encoding of opaque<> and string<>
are identical, the distinction being solely in interpretation. Opaque
data should not be interpreted but can be compared bytewise to other
opaque data. All strings use the UTF-8 encoding.
### Index (Type = 1)
The Index message defines the contents of the senders repository. A Index
message is sent by each peer immediately upon connection and whenever the
local repository contents changes. However, if a peer has no data to
advertise (the repository is empty, or it is set to only import data) it
is allowed but not required to send an empty Index message (a file list of
zero length). If the repository contents change from non-empty to empty,
an empty Index message must be sent. There is no response to the Index
message.
The Index message defines the contents of the senders repository. An
Index message is sent by each peer immediately upon connection. A peer
with no data to advertise (the repository is empty, or it is set to only
import data) is allowed but not required to send an empty Index message
(a file list of zero length). If the repository contents change from
non-empty to empty, an empty Index message must be sent. There is no
response to the Index message.
struct IndexMessage {
string Repository<>;
FileInfo Files<>;
}
#### Graphical Representation
struct FileInfo {
string Name<>;
unsigned int Flags;
hyper Modified;
unsigned int Version;
BlockInfo Blocks<>;
}
IndexMessage Structure:
struct BlockInfo {
unsigned int Length;
opaque Hash<>
}
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Repository |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Repository (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Files |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more FileInfo Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
FileInfo Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Name |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Name (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flags |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Modified (64 bits) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Blocks |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more BlockInfo Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
BlockInfo Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Hash |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Hash (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#### Fields
The Repository field identifies the repository that the index message
pertains to. For single repository implementations an empty repository
ID is acceptable.
ID is acceptable, or the word "default". The Name is the file name path
relative to the repository root. The combination of Repository and Name
uniquely identifies each file in a cluster.
The file name is the part relative to the repository root. The
modification time is expressed as the number of seconds since the Unix
Epoch. The version field is a counter that increments each time the file
changes but resets to zero each time the modification is updated. This
is used to signal changes to the file (or file metadata) while the
modification time remains unchanged. The hash algorithm is implied by
the hash length. Currently, the hash must be 32 bytes long and computed
by SHA256.
The Version field is a counter that is initially zero for each file. It
is incremented each time a change is detected. The combination of
Repository, Name and Version uniquely identifies the contents of a file
at a certain point in time.
The flags field is made up of the following single bit flags:
The Flags field is made up of the following single bit flags:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
@@ -136,62 +193,128 @@ The flags field is made up of the following single bit flags:
- Bit 0 through 17 are reserved for future use and shall be set to
zero.
The hash algorithm is implied by the Hash length. Currently, the hash
must be 32 bytes long and computed by SHA256.
The Modified time is expressed as the number of seconds since the Unix
Epoch. In the rare occasion that a file is simultaneously and
independently modified by two nodes in the same cluster and thus end up
on the same Version number after modification, the Modified field is
used as a tie breaker.
The Size field is the size of the file, in bytes.
The Blocks list contains the size and hash for each block in the file.
Each block represents a 128 KiB slice of the file, except for the last
block which may represent a smaller amount of data.
#### XDR
struct IndexMessage {
string Repository<>;
FileInfo Files<>;
}
struct FileInfo {
string Name<>;
unsigned int Flags;
hyper Modified;
unsigned int Version;
BlockInfo Blocks<>;
}
struct BlockInfo {
unsigned int Size;
opaque Hash<>;
}
### Request (Type = 2)
The Request message expresses the desire to receive a data block
corresponding to a part of a certain file in the peer's repository.
The requested block must correspond exactly to one block seen in the
peer's Index message. The hash field must be set to the expected value by
the sender. The receiver may validate that this is actually the case
before transmitting data. Each Request message must be met with a Response
#### Graphical Representation
RequestMessage Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Repository |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Repository (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Name |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Name (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Offset (64 bits) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#### Fields
The Repository and Name fields are as documented for the Index message.
The Offset and Size fields specify the region of the file to be
transferred. This should equate to exactly one block as seen in an Index
message.
#### XDR
struct RequestMessage {
string Repository<>;
string Name<>;
unsigned hyper Offset;
unsigned int Length;
opaque Hash<>;
unsigned int Size;
}
The hash algorithm is implied by the hash length. Currently, the hash
must be 32 bytes long and computed by SHA256.
The Message ID in the header must set to a unique value to be able to
correlate the request with the response message.
### Response (Type = 3)
The Response message is sent in response to a Request message. In case the
requested data was not available (an outdated block was requested, or
the file has been deleted), the Data field is empty.
The Response message is sent in response to a Request message.
#### Graphical Representation
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Data (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#### Fields
The Data field contains either a full 128 KiB block, a shorter block in
the case of the last block in a file, or is empty (zero length) if the
requested block is not available.
#### XDR
struct ResponseMessage {
opaque Data<>
}
The Message ID in the header is used to correlate requests and
responses.
### Ping (Type = 4)
The Ping message is used to determine that a connection is alive, and to
keep connections alive through state tracking network elements such as
firewalls and NAT gateways. The Ping message has no contents.
struct PingMessage {
}
### Pong (Type = 5)
The Pong message is sent in response to a Ping. The Pong message has no
contents, but copies the Message ID from the Ping.
struct PongMessage {
}
### IndexUpdate (Type = 6)
### Index Update (Type = 6)
This message has exactly the same structure as the Index message.
However instead of replacing the contents of the repository in the
@@ -206,26 +329,59 @@ configuration, version, etc. It is sent at connection initiation and,
optionally, when any of the sent parameters have changed. The message is
in the form of a list of (key, value) pairs, both of string type.
Key ID:s apart from the well known ones are implementation specific. An
implementation is expected to ignore unknown keys. An implementation may
impose limits on key and value size.
Well known keys:
- "clientId" -- The name of the implementation. Example: "syncthing".
- "clientVersion" -- The version of the client. Example: "v1.0.33-47". The
Following the SemVer 2.0 specification for version strings is
encouraged but not enforced.
#### Graphical Representation
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of Options |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Zero or more KeyValue Structures \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
KeyValue Structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Key |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Key (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of Value |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
\ Value (variable length) \
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#### XDR
struct OptionsMessage {
KeyValue Options<>;
}
struct KeyValue {
string Key;
string Value;
string Key<>;
string Value<>;
}
Key ID:s apart from the well known ones are implementation
specific. An implementation is expected to ignore unknown keys. An
implementation may impose limits on key and value size.
Well known keys:
- "clientId" -- The name of the implementation. Example: "syncthing".
- "clientVersion" -- The version of the client. Example: "v1.0.33-47". The
Following the SemVer 2.0 specification for version strings is
encouraged but not enforced.
Example Exchange
----------------
@@ -239,7 +395,7 @@ Example Exchange
7. <-Response
8. <-Response
9. <-Response
10. Index->
10. Index Update->
...
11. Ping->
12. <-Pong
@@ -250,7 +406,7 @@ of the data in the cluster. In this example, peer A has four missing or
outdated blocks. At 2 through 5 peer A sends requests for these blocks.
The requests are received by peer B, who retrieves the data from the
repository and transmits Response records (6 through 9). Node A updates
their repository contents and transmits an updated Index message (10).
their repository contents and transmits an Index Update message (10).
Both peers enter idle state after 10. At some later time 11, peer A
determines that it has not seen data from B for some time and sends a
Ping request. A response is sent at 12.

34
protocol/header.go Normal file
View File

@@ -0,0 +1,34 @@
package protocol
import "github.com/calmh/syncthing/xdr"
type header struct {
version int
msgID int
msgType int
}
func (h header) encodeXDR(xw *xdr.Writer) (int, error) {
u := encodeHeader(h)
return xw.WriteUint32(u)
}
func (h *header) decodeXDR(xr *xdr.Reader) error {
u := xr.ReadUint32()
*h = decodeHeader(u)
return xr.Error()
}
func encodeHeader(h header) uint32 {
return uint32(h.version&0xf)<<28 +
uint32(h.msgID&0xfff)<<16 +
uint32(h.msgType&0xff)<<8
}
func decodeHeader(u uint32) header {
return header{
version: int(u>>28) & 0xf,
msgID: int(u>>16) & 0xfff,
msgType: int(u>>8) & 0xff,
}
}

35
protocol/message_types.go Normal file
View File

@@ -0,0 +1,35 @@
package protocol
type IndexMessage struct {
Repository string // max:64
Files []FileInfo // max:100000
}
type FileInfo struct {
Name string // max:1024
Flags uint32
Modified int64
Version uint32
Blocks []BlockInfo // max:100000
}
type BlockInfo struct {
Size uint32
Hash []byte // max:64
}
type RequestMessage struct {
Repository string // max:64
Name string // max:1024
Offset uint64
Size uint32
}
type OptionsMessage struct {
Options []Option // max:64
}
type Option struct {
Key string // max:64
Value string // max:1024
}

286
protocol/message_xdr.go Normal file
View File

@@ -0,0 +1,286 @@
package protocol
import (
"bytes"
"io"
"github.com/calmh/syncthing/xdr"
)
func (o IndexMessage) EncodeXDR(w io.Writer) (int, error) {
var xw = xdr.NewWriter(w)
return o.encodeXDR(xw)
}
func (o IndexMessage) MarshalXDR() []byte {
var buf bytes.Buffer
var xw = xdr.NewWriter(&buf)
o.encodeXDR(xw)
return buf.Bytes()
}
func (o IndexMessage) encodeXDR(xw *xdr.Writer) (int, error) {
if len(o.Repository) > 64 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteString(o.Repository)
if len(o.Files) > 100000 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteUint32(uint32(len(o.Files)))
for i := range o.Files {
o.Files[i].encodeXDR(xw)
}
return xw.Tot(), xw.Error()
}
func (o *IndexMessage) DecodeXDR(r io.Reader) error {
xr := xdr.NewReader(r)
return o.decodeXDR(xr)
}
func (o *IndexMessage) UnmarshalXDR(bs []byte) error {
var buf = bytes.NewBuffer(bs)
var xr = xdr.NewReader(buf)
return o.decodeXDR(xr)
}
func (o *IndexMessage) decodeXDR(xr *xdr.Reader) error {
o.Repository = xr.ReadStringMax(64)
_FilesSize := int(xr.ReadUint32())
if _FilesSize > 100000 {
return xdr.ErrElementSizeExceeded
}
o.Files = make([]FileInfo, _FilesSize)
for i := range o.Files {
(&o.Files[i]).decodeXDR(xr)
}
return xr.Error()
}
func (o FileInfo) EncodeXDR(w io.Writer) (int, error) {
var xw = xdr.NewWriter(w)
return o.encodeXDR(xw)
}
func (o FileInfo) MarshalXDR() []byte {
var buf bytes.Buffer
var xw = xdr.NewWriter(&buf)
o.encodeXDR(xw)
return buf.Bytes()
}
func (o FileInfo) encodeXDR(xw *xdr.Writer) (int, error) {
if len(o.Name) > 1024 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteString(o.Name)
xw.WriteUint32(o.Flags)
xw.WriteUint64(uint64(o.Modified))
xw.WriteUint32(o.Version)
if len(o.Blocks) > 100000 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteUint32(uint32(len(o.Blocks)))
for i := range o.Blocks {
o.Blocks[i].encodeXDR(xw)
}
return xw.Tot(), xw.Error()
}
func (o *FileInfo) DecodeXDR(r io.Reader) error {
xr := xdr.NewReader(r)
return o.decodeXDR(xr)
}
func (o *FileInfo) UnmarshalXDR(bs []byte) error {
var buf = bytes.NewBuffer(bs)
var xr = xdr.NewReader(buf)
return o.decodeXDR(xr)
}
func (o *FileInfo) decodeXDR(xr *xdr.Reader) error {
o.Name = xr.ReadStringMax(1024)
o.Flags = xr.ReadUint32()
o.Modified = int64(xr.ReadUint64())
o.Version = xr.ReadUint32()
_BlocksSize := int(xr.ReadUint32())
if _BlocksSize > 100000 {
return xdr.ErrElementSizeExceeded
}
o.Blocks = make([]BlockInfo, _BlocksSize)
for i := range o.Blocks {
(&o.Blocks[i]).decodeXDR(xr)
}
return xr.Error()
}
func (o BlockInfo) EncodeXDR(w io.Writer) (int, error) {
var xw = xdr.NewWriter(w)
return o.encodeXDR(xw)
}
func (o BlockInfo) MarshalXDR() []byte {
var buf bytes.Buffer
var xw = xdr.NewWriter(&buf)
o.encodeXDR(xw)
return buf.Bytes()
}
func (o BlockInfo) encodeXDR(xw *xdr.Writer) (int, error) {
xw.WriteUint32(o.Size)
if len(o.Hash) > 64 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteBytes(o.Hash)
return xw.Tot(), xw.Error()
}
func (o *BlockInfo) DecodeXDR(r io.Reader) error {
xr := xdr.NewReader(r)
return o.decodeXDR(xr)
}
func (o *BlockInfo) UnmarshalXDR(bs []byte) error {
var buf = bytes.NewBuffer(bs)
var xr = xdr.NewReader(buf)
return o.decodeXDR(xr)
}
func (o *BlockInfo) decodeXDR(xr *xdr.Reader) error {
o.Size = xr.ReadUint32()
o.Hash = xr.ReadBytesMax(64)
return xr.Error()
}
func (o RequestMessage) EncodeXDR(w io.Writer) (int, error) {
var xw = xdr.NewWriter(w)
return o.encodeXDR(xw)
}
func (o RequestMessage) MarshalXDR() []byte {
var buf bytes.Buffer
var xw = xdr.NewWriter(&buf)
o.encodeXDR(xw)
return buf.Bytes()
}
func (o RequestMessage) encodeXDR(xw *xdr.Writer) (int, error) {
if len(o.Repository) > 64 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteString(o.Repository)
if len(o.Name) > 1024 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteString(o.Name)
xw.WriteUint64(o.Offset)
xw.WriteUint32(o.Size)
return xw.Tot(), xw.Error()
}
func (o *RequestMessage) DecodeXDR(r io.Reader) error {
xr := xdr.NewReader(r)
return o.decodeXDR(xr)
}
func (o *RequestMessage) UnmarshalXDR(bs []byte) error {
var buf = bytes.NewBuffer(bs)
var xr = xdr.NewReader(buf)
return o.decodeXDR(xr)
}
func (o *RequestMessage) decodeXDR(xr *xdr.Reader) error {
o.Repository = xr.ReadStringMax(64)
o.Name = xr.ReadStringMax(1024)
o.Offset = xr.ReadUint64()
o.Size = xr.ReadUint32()
return xr.Error()
}
func (o OptionsMessage) EncodeXDR(w io.Writer) (int, error) {
var xw = xdr.NewWriter(w)
return o.encodeXDR(xw)
}
func (o OptionsMessage) MarshalXDR() []byte {
var buf bytes.Buffer
var xw = xdr.NewWriter(&buf)
o.encodeXDR(xw)
return buf.Bytes()
}
func (o OptionsMessage) encodeXDR(xw *xdr.Writer) (int, error) {
if len(o.Options) > 64 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteUint32(uint32(len(o.Options)))
for i := range o.Options {
o.Options[i].encodeXDR(xw)
}
return xw.Tot(), xw.Error()
}
func (o *OptionsMessage) DecodeXDR(r io.Reader) error {
xr := xdr.NewReader(r)
return o.decodeXDR(xr)
}
func (o *OptionsMessage) UnmarshalXDR(bs []byte) error {
var buf = bytes.NewBuffer(bs)
var xr = xdr.NewReader(buf)
return o.decodeXDR(xr)
}
func (o *OptionsMessage) decodeXDR(xr *xdr.Reader) error {
_OptionsSize := int(xr.ReadUint32())
if _OptionsSize > 64 {
return xdr.ErrElementSizeExceeded
}
o.Options = make([]Option, _OptionsSize)
for i := range o.Options {
(&o.Options[i]).decodeXDR(xr)
}
return xr.Error()
}
func (o Option) EncodeXDR(w io.Writer) (int, error) {
var xw = xdr.NewWriter(w)
return o.encodeXDR(xw)
}
func (o Option) MarshalXDR() []byte {
var buf bytes.Buffer
var xw = xdr.NewWriter(&buf)
o.encodeXDR(xw)
return buf.Bytes()
}
func (o Option) encodeXDR(xw *xdr.Writer) (int, error) {
if len(o.Key) > 64 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteString(o.Key)
if len(o.Value) > 1024 {
return xw.Tot(), xdr.ErrElementSizeExceeded
}
xw.WriteString(o.Value)
return xw.Tot(), xw.Error()
}
func (o *Option) DecodeXDR(r io.Reader) error {
xr := xdr.NewReader(r)
return o.decodeXDR(xr)
}
func (o *Option) UnmarshalXDR(bs []byte) error {
var buf = bytes.NewBuffer(bs)
var xr = xdr.NewReader(buf)
return o.decodeXDR(xr)
}
func (o *Option) decodeXDR(xr *xdr.Reader) error {
o.Key = xr.ReadStringMax(64)
o.Value = xr.ReadStringMax(1024)
return xr.Error()
}

186
protocol/messages.go
View File

@@ -1,186 +0,0 @@
package protocol
import (
"errors"
"io"
"github.com/calmh/syncthing/buffers"
"github.com/calmh/syncthing/xdr"
)
const (
maxNumFiles = 100000 // More than 100000 files is a protocol error
maxNumBlocks = 100000 // 100000 * 128KB = 12.5 GB max acceptable file size
)
var (
ErrMaxFilesExceeded = errors.New("Protocol error: number of files per index exceeds limit")
ErrMaxBlocksExceeded = errors.New("Protocol error: number of blocks per file exceeds limit")
)
type request struct {
repo string
name string
offset int64
size uint32
hash []byte
}
type header struct {
version int
msgID int
msgType int
}
func encodeHeader(h header) uint32 {
return uint32(h.version&0xf)<<28 +
uint32(h.msgID&0xfff)<<16 +
uint32(h.msgType&0xff)<<8
}
func decodeHeader(u uint32) header {
return header{
version: int(u>>28) & 0xf,
msgID: int(u>>16) & 0xfff,
msgType: int(u>>8) & 0xff,
}
}
func WriteIndex(w io.Writer, repo string, idx []FileInfo) (int, error) {
mw := newMarshalWriter(w)
mw.writeIndex(repo, idx)
return int(mw.Tot()), mw.Err()
}
type marshalWriter struct {
*xdr.Writer
}
func newMarshalWriter(w io.Writer) marshalWriter {
return marshalWriter{xdr.NewWriter(w)}
}
func (w *marshalWriter) writeHeader(h header) {
w.WriteUint32(encodeHeader(h))
}
func (w *marshalWriter) writeIndex(repo string, idx []FileInfo) {
w.WriteString(repo)
w.WriteUint32(uint32(len(idx)))
for _, f := range idx {
w.WriteString(f.Name)
w.WriteUint32(f.Flags)
w.WriteUint64(uint64(f.Modified))
w.WriteUint32(f.Version)
w.WriteUint32(uint32(len(f.Blocks)))
for _, b := range f.Blocks {
w.WriteUint32(b.Size)
w.WriteBytes(b.Hash)
}
}
}
func (w *marshalWriter) writeRequest(r request) {
w.WriteString(r.repo)
w.WriteString(r.name)
w.WriteUint64(uint64(r.offset))
w.WriteUint32(r.size)
w.WriteBytes(r.hash)
}
func (w *marshalWriter) writeResponse(data []byte) {
w.WriteBytes(data)
}
func (w *marshalWriter) writeOptions(opts map[string]string) {
w.WriteUint32(uint32(len(opts)))
for k, v := range opts {
w.WriteString(k)
w.WriteString(v)
}
}
func ReadIndex(r io.Reader) (string, []FileInfo, error) {
mr := newMarshalReader(r)
repo, idx := mr.readIndex()
return repo, idx, mr.Err()
}
type marshalReader struct {
*xdr.Reader
err error
}
func newMarshalReader(r io.Reader) marshalReader {
return marshalReader{
Reader: xdr.NewReader(r),
err: nil,
}
}
func (r marshalReader) Err() error {
if r.err != nil {
return r.err
}
return r.Reader.Err()
}
func (r marshalReader) readHeader() header {
return decodeHeader(r.ReadUint32())
}
func (r marshalReader) readIndex() (string, []FileInfo) {
var files []FileInfo
repo := r.ReadString()
nfiles := r.ReadUint32()
if nfiles > maxNumFiles {
r.err = ErrMaxFilesExceeded
return "", nil
}
if nfiles > 0 {
files = make([]FileInfo, nfiles)
for i := range files {
files[i].Name = r.ReadString()
files[i].Flags = r.ReadUint32()
files[i].Modified = int64(r.ReadUint64())
files[i].Version = r.ReadUint32()
nblocks := r.ReadUint32()
if nblocks > maxNumBlocks {
r.err = ErrMaxBlocksExceeded
return "", nil
}
blocks := make([]BlockInfo, nblocks)
for j := range blocks {
blocks[j].Size = r.ReadUint32()
blocks[j].Hash = r.ReadBytes(buffers.Get(32))
}
files[i].Blocks = blocks
}
}
return repo, files
}
func (r marshalReader) readRequest() request {
var req request
req.repo = r.ReadString()
req.name = r.ReadString()
req.offset = int64(r.ReadUint64())
req.size = r.ReadUint32()
req.hash = r.ReadBytes(buffers.Get(32))
return req
}
func (r marshalReader) readResponse() []byte {
return r.ReadBytes(buffers.Get(128 * 1024))
}
func (r marshalReader) readOptions() map[string]string {
n := r.ReadUint32()
opts := make(map[string]string, n)
for i := 0; i < int(n); i++ {
k := r.ReadString()
v := r.ReadString()
opts[k] = v
}
return opts
}

143
protocol/messages_test.go
View File

@@ -1,143 +0,0 @@
package protocol
import (
"bytes"
"io/ioutil"
"reflect"
"testing"
"testing/quick"
)
func TestIndex(t *testing.T) {
idx := []FileInfo{
{
"Foo",
FlagInvalid & FlagDeleted & 0755,
1234567890,
142,
[]BlockInfo{
{12345678, []byte("hash hash hash")},
{23456781, []byte("ash hash hashh")},
{34567812, []byte("sh hash hashha")},
},
}, {
"Quux/Quux",
0644,
2345678901,
232323232,
[]BlockInfo{
{45678123, []byte("4321 hash hash hash")},
{56781234, []byte("3214 ash hash hashh")},
{67812345, []byte("2143 sh hash hashha")},
},
},
}
var buf = new(bytes.Buffer)
var wr = newMarshalWriter(buf)
wr.writeIndex("default", idx)
var rd = newMarshalReader(buf)
var repo, idx2 = rd.readIndex()
if repo != "default" {
t.Error("Incorrect repo", repo)
}
if !reflect.DeepEqual(idx, idx2) {
t.Errorf("Index marshal error:\n%#v\n%#v\n", idx, idx2)
}
}
func TestRequest(t *testing.T) {
f := func(repo, name string, offset int64, size uint32, hash []byte) bool {
var buf = new(bytes.Buffer)
var req = request{repo, name, offset, size, hash}
var wr = newMarshalWriter(buf)
wr.writeRequest(req)
var rd = newMarshalReader(buf)
var req2 = rd.readRequest()
return req.name == req2.name &&
req.offset == req2.offset &&
req.size == req2.size &&
bytes.Compare(req.hash, req2.hash) == 0
}
if err := quick.Check(f, nil); err != nil {
t.Error(err)
}
}
func TestResponse(t *testing.T) {
f := func(data []byte) bool {
var buf = new(bytes.Buffer)
var wr = newMarshalWriter(buf)
wr.writeResponse(data)
var rd = newMarshalReader(buf)
var read = rd.readResponse()
return bytes.Compare(read, data) == 0
}
if err := quick.Check(f, nil); err != nil {
t.Error(err)
}
}
func BenchmarkWriteIndex(b *testing.B) {
idx := []FileInfo{
{
"Foo",
0777,
1234567890,
424242,
[]BlockInfo{
{12345678, []byte("hash hash hash")},
{23456781, []byte("ash hash hashh")},
{34567812, []byte("sh hash hashha")},
},
}, {
"Quux/Quux",
0644,
2345678901,
323232,
[]BlockInfo{
{45678123, []byte("4321 hash hash hash")},
{56781234, []byte("3214 ash hash hashh")},
{67812345, []byte("2143 sh hash hashha")},
},
},
}
var wr = newMarshalWriter(ioutil.Discard)
for i := 0; i < b.N; i++ {
wr.writeIndex("default", idx)
}
}
func BenchmarkWriteRequest(b *testing.B) {
var req = request{"default", "blah blah", 1231323, 13123123, []byte("hash hash hash")}
var wr = newMarshalWriter(ioutil.Discard)
for i := 0; i < b.N; i++ {
wr.writeRequest(req)
}
}
func TestOptions(t *testing.T) {
opts := map[string]string{
"foo": "bar",
"someKey": "otherValue",
"hello": "",
"": "42",
}
var buf = new(bytes.Buffer)
var wr = newMarshalWriter(buf)
wr.writeOptions(opts)
var rd = newMarshalReader(buf)
var ropts = rd.readOptions()
if !reflect.DeepEqual(opts, ropts) {
t.Error("Incorrect options marshal/demarshal")
}
}

157
protocol/protocol.go
View File

@@ -13,6 +13,8 @@ import (
"github.com/calmh/syncthing/xdr"
)
const BlockSize = 128 * 1024
const (
messageTypeIndex = 1
messageTypeRequest = 2
@@ -32,26 +34,13 @@ var (
ErrClusterHash = fmt.Errorf("Configuration error: mismatched cluster hash")
)
type FileInfo struct {
Name string
Flags uint32
Modified int64
Version uint32
Blocks []BlockInfo
}
type BlockInfo struct {
Size uint32
Hash []byte
}
type Model interface {
// An index was received from the peer node
Index(nodeID string, files []FileInfo)
// An index update was received from the peer node
IndexUpdate(nodeID string, files []FileInfo)
// A request was made by the peer node
Request(nodeID, repo string, name string, offset int64, size uint32, hash []byte) ([]byte, error)
Request(nodeID, repo string, name string, offset int64, size int) ([]byte, error)
// The peer node closed the connection
Close(nodeID string, err error)
}
@@ -62,9 +51,9 @@ type Connection struct {
id string
receiver Model
reader io.Reader
mreader marshalReader
xr *xdr.Reader
writer io.Writer
mwriter marshalWriter
xw *xdr.Writer
closed bool
awaiting map[int]chan asyncResult
nextId int
@@ -102,9 +91,9 @@ func NewConnection(nodeID string, reader io.Reader, writer io.Writer, receiver M
id: nodeID,
receiver: receiver,
reader: flrd,
mreader: marshalReader{Reader: xdr.NewReader(flrd)},
xr: xdr.NewReader(flrd),
writer: flwr,
mwriter: marshalWriter{Writer: xdr.NewWriter(flwr)},
xw: xdr.NewWriter(flwr),
awaiting: make(map[int]chan asyncResult),
indexSent: make(map[string]map[string][2]int64),
}
@@ -116,9 +105,16 @@ func NewConnection(nodeID string, reader io.Reader, writer io.Writer, receiver M
c.myOptions = options
go func() {
c.Lock()
c.mwriter.writeHeader(header{0, c.nextId, messageTypeOptions})
c.mwriter.writeOptions(options)
err := c.flush()
header{0, c.nextId, messageTypeOptions}.encodeXDR(c.xw)
var om OptionsMessage
for k, v := range options {
om.Options = append(om.Options, Option{k, v})
}
om.encodeXDR(c.xw)
err := c.xw.Error()
if err == nil {
err = c.flush()
}
if err != nil {
log.Println("Warning: Write error during initial handshake:", err)
}
@@ -159,9 +155,11 @@ func (c *Connection) Index(repo string, idx []FileInfo) {
idx = diff
}
c.mwriter.writeHeader(header{0, c.nextId, msgType})
c.mwriter.writeIndex(repo, idx)
err := c.flush()
header{0, c.nextId, msgType}.encodeXDR(c.xw)
_, err := IndexMessage{repo, idx}.encodeXDR(c.xw)
if err == nil {
err = c.flush()
}
c.nextId = (c.nextId + 1) & 0xfff
c.hasSentIndex = true
c.Unlock()
@@ -169,14 +167,11 @@ func (c *Connection) Index(repo string, idx []FileInfo) {
if err != nil {
c.close(err)
return
} else if c.mwriter.Err() != nil {
c.close(c.mwriter.Err())
return
}
}
// Request returns the bytes for the specified block after fetching them from the connected peer.
func (c *Connection) Request(repo string, name string, offset int64, size uint32, hash []byte) ([]byte, error) {
func (c *Connection) Request(repo string, name string, offset int64, size int) ([]byte, error) {
c.Lock()
if c.closed {
c.Unlock()
@@ -184,14 +179,11 @@ func (c *Connection) Request(repo string, name string, offset int64, size uint32
}
rc := make(chan asyncResult)
c.awaiting[c.nextId] = rc
c.mwriter.writeHeader(header{0, c.nextId, messageTypeRequest})
c.mwriter.writeRequest(request{repo, name, offset, size, hash})
if c.mwriter.Err() != nil {
c.Unlock()
c.close(c.mwriter.Err())
return nil, c.mwriter.Err()
header{0, c.nextId, messageTypeRequest}.encodeXDR(c.xw)
_, err := RequestMessage{repo, name, uint64(offset), uint32(size)}.encodeXDR(c.xw)
if err == nil {
err = c.flush()
}
err := c.flush()
if err != nil {
c.Unlock()
c.close(err)
@@ -215,15 +207,15 @@ func (c *Connection) ping() bool {
}
rc := make(chan asyncResult, 1)
c.awaiting[c.nextId] = rc
c.mwriter.writeHeader(header{0, c.nextId, messageTypePing})
header{0, c.nextId, messageTypePing}.encodeXDR(c.xw)
err := c.flush()
if err != nil {
c.Unlock()
c.close(err)
return false
} else if c.mwriter.Err() != nil {
} else if c.xw.Error() != nil {
c.Unlock()
c.close(c.mwriter.Err())
c.close(c.xw.Error())
return false
}
c.nextId = (c.nextId + 1) & 0xfff
@@ -269,9 +261,10 @@ func (c *Connection) isClosed() bool {
func (c *Connection) readerLoop() {
loop:
for {
hdr := c.mreader.readHeader()
if c.mreader.Err() != nil {
c.close(c.mreader.Err())
var hdr header
hdr.decodeXDR(c.xr)
if c.xr.Error() != nil {
c.close(c.xr.Error())
break loop
}
if hdr.version != 0 {
@@ -281,64 +274,65 @@ loop:
switch hdr.msgType {
case messageTypeIndex:
repo, files := c.mreader.readIndex()
_ = repo
if c.mreader.Err() != nil {
c.close(c.mreader.Err())
var im IndexMessage
im.decodeXDR(c.xr)
if c.xr.Error() != nil {
c.close(c.xr.Error())
break loop
} else {
c.receiver.Index(c.id, files)
c.receiver.Index(c.id, im.Files)
}
c.Lock()
c.hasRecvdIndex = true
c.Unlock()
case messageTypeIndexUpdate:
repo, files := c.mreader.readIndex()
_ = repo
if c.mreader.Err() != nil {
c.close(c.mreader.Err())
var im IndexMessage
im.decodeXDR(c.xr)
if c.xr.Error() != nil {
c.close(c.xr.Error())
break loop
} else {
c.receiver.IndexUpdate(c.id, files)
c.receiver.IndexUpdate(c.id, im.Files)
}
case messageTypeRequest:
req := c.mreader.readRequest()
if c.mreader.Err() != nil {
c.close(c.mreader.Err())
var req RequestMessage
req.decodeXDR(c.xr)
if c.xr.Error() != nil {
c.close(c.xr.Error())
break loop
}
go c.processRequest(hdr.msgID, req)
case messageTypeResponse:
data := c.mreader.readResponse()
data := c.xr.ReadBytes()
if c.mreader.Err() != nil {
c.close(c.mreader.Err())
if c.xr.Error() != nil {
c.close(c.xr.Error())
break loop
} else {
c.Lock()
rc, ok := c.awaiting[hdr.msgID]
delete(c.awaiting, hdr.msgID)
c.Unlock()
}
if ok {
rc <- asyncResult{data, c.mreader.Err()}
close(rc)
}
c.Lock()
rc, ok := c.awaiting[hdr.msgID]
delete(c.awaiting, hdr.msgID)
c.Unlock()
if ok {
rc <- asyncResult{data, c.xr.Error()}
close(rc)
}
case messageTypePing:
c.Lock()
c.mwriter.WriteUint32(encodeHeader(header{0, hdr.msgID, messageTypePong}))
header{0, hdr.msgID, messageTypePong}.encodeXDR(c.xw)
err := c.flush()
c.Unlock()
if err != nil {
c.close(err)
break loop
} else if c.mwriter.Err() != nil {
c.close(c.mwriter.Err())
} else if c.xw.Error() != nil {
c.close(c.xw.Error())
break loop
}
@@ -357,8 +351,18 @@ loop:
}
case messageTypeOptions:
var om OptionsMessage
om.decodeXDR(c.xr)
if c.xr.Error() != nil {
c.close(c.xr.Error())
break loop
}
c.optionsLock.Lock()
c.peerOptions = c.mreader.readOptions()
c.peerOptions = make(map[string]string, len(om.Options))
for _, opt := range om.Options {
c.peerOptions[opt.Key] = opt.Value
}
c.optionsLock.Unlock()
if mh, rh := c.myOptions["clusterHash"], c.peerOptions["clusterHash"]; len(mh) > 0 && len(rh) > 0 && mh != rh {
@@ -373,13 +377,12 @@ loop:
}
}
func (c *Connection) processRequest(msgID int, req request) {
data, _ := c.receiver.Request(c.id, req.repo, req.name, req.offset, req.size, req.hash)
func (c *Connection) processRequest(msgID int, req RequestMessage) {
data, _ := c.receiver.Request(c.id, req.Repository, req.Name, int64(req.Offset), int(req.Size))
c.Lock()
c.mwriter.WriteUint32(encodeHeader(header{0, msgID, messageTypeResponse}))
c.mwriter.writeResponse(data)
err := c.mwriter.Err()
header{0, msgID, messageTypeResponse}.encodeXDR(c.xw)
_, err := c.xw.WriteBytes(data)
if err == nil {
err = c.flush()
}
@@ -428,8 +431,8 @@ func (c *Connection) Statistics() Statistics {
stats := Statistics{
At: time.Now(),
InBytesTotal: int(c.mreader.Tot()),
OutBytesTotal: int(c.mwriter.Tot()),
InBytesTotal: int(c.xr.Tot()),
OutBytesTotal: int(c.xw.Tot()),
}
return stats

19
protocol/protocol_test.go
View File

@@ -80,7 +80,7 @@ func TestRequestResponseErr(t *testing.T) {
NewConnection("c0", ar, ebw, m0, nil)
c1 := NewConnection("c1", br, eaw, m1, nil)
d, err := c1.Request("default", "tn", 1234, 3456, []byte("hashbytes"))
d, err := c1.Request("default", "tn", 1234)
if err == e || err == ErrClosed {
t.Logf("Error at %d+%d bytes", i, j)
if !m1.closed {
@@ -104,15 +104,12 @@ func TestRequestResponseErr(t *testing.T) {
if m0.name != "tn" {
t.Error("Incorrect name %q", m0.name)
}
if m0.offset != 1234 {
if m0.offset != 1234*BlockSize {
t.Error("Incorrect offset %d", m0.offset)
}
if m0.size != 3456 {
if m0.size != BlockSize {
t.Error("Incorrect size %d", m0.size)
}
if string(m0.hash) != "hashbytes" {
t.Error("Incorrect hash %q", m0.hash)
}
t.Logf("Pass at %d+%d bytes", i, j)
pass = true
}
@@ -132,11 +129,11 @@ func TestVersionErr(t *testing.T) {
c0 := NewConnection("c0", ar, bw, m0, nil)
NewConnection("c1", br, aw, m1, nil)
c0.mwriter.writeHeader(header{
c0.xw.WriteUint32(encodeHeader(header{
version: 2,
msgID: 0,
msgType: 0,
})
}))
c0.flush()
if !m1.closed {
@@ -154,11 +151,11 @@ func TestTypeErr(t *testing.T) {
c0 := NewConnection("c0", ar, bw, m0, nil)
NewConnection("c1", br, aw, m1, nil)
c0.mwriter.writeHeader(header{
c0.xw.WriteUint32(encodeHeader(header{
version: 0,
msgID: 0,
msgType: 42,
})
}))
c0.flush()
if !m1.closed {
@@ -193,7 +190,7 @@ func TestClose(t *testing.T) {
c0.Index("default", nil)
c0.Index("default", nil)
_, err := c0.Request("default", "foo", 0, 0, nil)
_, err := c0.Request("default", "foo", 0)
if err == nil {
t.Error("Request should return an error")
}