perso/syncthing-arm
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

vendor: Add dependencies for discosrv

Browse Source
This commit is contained in:
Jakob Borg
2016-05-31 22:35:35 +02:00
parent eacae83886
commit f9e2623fdc
126 changed files with 60401 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
vendor/github.com/cznic/fileutil/LICENSE generated vendored Normal file
View File

@@ -0,0 +1,27 @@
Copyright (c) 2014 The fileutil Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the names of the authors nor the names of the
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

251
vendor/github.com/cznic/fileutil/falloc/docs.go generated vendored Normal file
View File

@@ -0,0 +1,251 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
/*
WIP: Package falloc provides allocation/deallocation of space within a
file/store (WIP, unstable API).
Overall structure:
File == n blocks.
Block == n atoms.
Atom == 16 bytes.
x6..x0 == least significant 7 bytes of a 64 bit integer, highest (7th) byte is
0 and is not stored in the file.
Block first byte
Aka block type tag.
------------------------------------------------------------------------------
0xFF: Free atom (free block of size 1).
+------++---------++---------++------+
| 0 || 1...7 || 8...14 || 15 |
+------++---------++---------++------+
| 0xFF || p6...p0 || n6...n0 || 0xFF |
+------++---------++---------++------+
Link to the previous free block (atom addressed) is p6...p0, next dtto in
n6...n0. Doubly linked lists of "compatible" free blocks allows for free space
reclaiming and merging. "Compatible" == of size at least some K. Heads of all
such lists are organized per K or intervals of Ks elsewhere.
------------------------------------------------------------------------------
0xFE: Free block, size == s6...s0 atoms.
+------++---------++---------++---------++--
| +0 || 1...7 || 8...14 || 15...21 || 22...16*size-1
+------++---------++---------++---------++--
| 0xFE || p6...p0 || n6...n0 || s6...s0 || ...
+------++---------++---------++---------++--
Prev and next links as in the 0xFF first byte case. End of this block - see
"Block last byte": 0xFE bellow. Data between == undefined.
------------------------------------------------------------------------------
0xFD: Relocated block.
+------++---------++-----------++------+
| 0 || 1...7 || 8...14 || 15 |
+------++---------++-----------++------+
| 0xFD || r6...r0 || undefined || 0x00 | // == used block
+------++---------++-----------++------+
Relocation link is r6..r0 == atom address. Relocations MUST NOT chain and MUST
point to a "content" block, i.e. one with the first byte in 0x00...0xFC.
Relocated block allows to permanently assign a handle/file pointer ("atom"
address) to some content and resize the content anytime afterwards w/o having
to update all the possible existing references to the original handle.
------------------------------------------------------------------------------
0xFC: Used long block.
+------++---------++--------------------++---------+---+
| 0 || 1...2 || 3...N+2 || | |
+------++---------++--------------------++---------+---+
| 0xFC || n1...n0 || N bytes of content || padding | Z |
+------++---------++--------------------++---------+---+
This block type is used for content of length in N == 238...61680 bytes. N is
encoded as a 2 byte unsigned integer n1..n0 in network byte order. Values
bellow 238 are reserved, those content lengths are to be carried by the
0x00..0xFB block types.
1. n in 0x00EE...0xF0F0 is used for content under the same rules
as in the 0x01..0xED type.
2. If the last byte of the content is not the last byte of an atom then
the last byte of the block is 0x00.
3. If the last byte of the content IS the last byte of an atom:
3.1 If the last byte of content is in 0x00..0xFD then everything is OK.
3.2 If the last byte of content is 0xFE or 0xFF then the escape
via n > 0xF0F0 MUST be used AND the block's last byte is 0x00 or 0x01,
meaning value 0xFE and 0xFF respectively.
4. n in 0xF0F1...0xFFFF is like the escaped 0xEE..0xFB block.
N == 13 + 16(n - 0xF0F1).
Discussion of the padding and Z fields - see the 0x01..0xED block type.
------------------------------------------------------------------------------
0xEE...0xFB: Used escaped short block.
+---++----------------------++---+
| 0 || 1...N-1 || |
+---++----------------------++---+
| X || N-1 bytes of content || Z |
+---++----------------------++---+
N == 15 + 16(X - 0xEE). Z is the content last byte encoded as follows.
case Z == 0x00: The last byte of content is 0xFE
case Z == 0x01: The last byte of content is 0xFF
------------------------------------------------------------------------------
0x01...0xED: Used short block.
+---++--------------------++---------+---+
| 0 || 1...N || | |
+---++--------------------++---------+---+
| N || N bytes of content || padding | Z |
+---++--------------------++---------+---+
This block type is used for content of length in 1...237 bytes. The value of
the "padding" field, if of non zero length, is undefined.
If the last byte of content is the last byte of an atom (== its file byte
offset & 0xF == 0xF) then such last byte MUST be in 0x00...0xFD.
If the last byte of content is the last byte of an atom AND the last byte of
content is 0xFE or 0xFF then the short escape block type (0xEE...0xFB) MUST be
used.
If the last byte of content is not the last byte of an atom, then the last byte
of such block, i.e. the Z field, which is also a last byte of some atom, MUST
be 0x00 (i.e. the used block marker). Other "tail" values are reserved.
------------------------------------------------------------------------------
0x00: Used empty block.
+------++-----------++------+
| 0 || 1...14 || 15 |
+------++-----------++------+
| 0x00 || undefined || 0x00 | // == used block, other "tail" values reserved.
+------++-----------++------+
All of the rules for 0x01..0xED applies. Depicted only for its different
semantics (e.g. an allocated [existing] string but with length of zero).
==============================================================================
Block last byte
------------------------------------------------------------------------------
0xFF: Free atom. Layout - see "Block first byte": FF.
------------------------------------------------------------------------------
0xFE: Free block, size n atoms. Preceding 7 bytes == size (s6...s0) of the free
block in atoms, network byte order
--++---------++------+
|| -8...-2 || -1 |
--++---------++------+
... || s6...s0 || 0xFE | <- block's last byte
--++---------++------+
Layout at start of this block - see "Block first byte": FE.
------------------------------------------------------------------------------
0x00...0xFD: Used (non free) block.
==============================================================================
Free lists table
The free lists table content is stored in the standard layout of a used block.
A table item is a 7 byte size field followed by a 7 byte atom address field
(both in network byte order), thus every item is 14 contiguous bytes. The
item's address field is pointing to a free block. The size field determines
the minimal size (in atoms) of free blocks on that list.
The free list table is n above items, thus the content has 14n bytes. Note that
the largest block content is 61680 bytes and as there are 14 bytes per table
item, so the table is limited to at most 4405 entries.
Items in the table do not have to be sorted according to their size field values.
No two items can have the same value of the size field.
When freeing blocks, the block MUST be linked into an item list with the
highest possible size field, which is less or equal to the number of atoms in
the new free block.
When freeing a block, the block MUST be first merged with any adjacent free
blocks (thus possibly creating a bigger free block) using information derived
from the adjacent blocks first and last bytes. Such merged free blocks MUST be
removed from their original doubly linked lists. Afterwards the new bigger free
block is put to the free list table in the appropriate item.
Items with address field == 0 are legal. Such item is a placeholder for a empty
list of free blocks of the item's size.
Items with size field == 0 are legal. Such item is a placeholder, used e.g. to
avoid further reallocations/redirecting of the free lists table.
The largest possible allocation request (for content length 61680 bytes) is
0xF10 (3856) atoms. All free blocks of this or bigger size are presumably put
into a single table item with the size 3856. It may be useful to additionally
have a free lists table item which links free blocks of some bigger size (say
1M+) and then use the OS sparse file support (if present) to save the physical
space used by such free blocks.
Smaller (<3856 atoms) free blocks can be organized exactly (every distinct size
has its table item) or the sizes can run using other schema like e.g. "1, 2,
4, 8, ..." (powers of 2) or "1, 2, 3, 5, 8, 13, ..." (the Fibonacci sequence)
or they may be fine tuned to a specific usage pattern.
==============================================================================
Header
The first block of a file (atom address == file offset == 0) is the file header.
The header block has the standard layout of a used short non escaped block.
Special conditions apply: The header block and its content MUST be like this:
+------+---------+---------+------+
| 0 | 1...7 | 8...14 | 15 |
+------+---------+---------+------+
| 0x0F | m6...m0 | f6...f0 | FLTT |
+------+---------+---------+------+
m6..m0 is a "magic" value 0xF1C1A1FE51B1E.
f6...f0 is the atom address of the free lists table (discussed elsewhere).
If f6...f0 == 0x00 the there is no free lists table (yet).
FLTT describes the type of the Free List Table. Currently defined values:
------------------------------------------------------------------------------
FLTT == 0: Free List Table is fixed at atom address 2. It has a fixed size for 3856 entries
for free list of size 1..3855 atoms and the last is for the list of free block >= 3856 atoms.
*/
package falloc
const (
INVALID_HANDLE = Handle(-1)
)

130
vendor/github.com/cznic/fileutil/falloc/error.go generated vendored Normal file
View File

@@ -0,0 +1,130 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package falloc
import "fmt"
// EBadRequest is an error produced for invalid operation, e.g. for data of more than maximum allowed.
type EBadRequest struct {
Name string
Size int
}
func (e *EBadRequest) Error() string {
return fmt.Sprintf("%s: size %d", e.Name, e.Size)
}
// EClose is a file/store close error.
type EClose struct {
Name string
Err error
}
func (e *EClose) Error() string {
return fmt.Sprintf("%sx: %s", e.Name, e.Err)
}
// ECorrupted is a file/store format error.
type ECorrupted struct {
Name string
Ofs int64
}
func (e *ECorrupted) Error() string {
return fmt.Sprintf("%s: corrupted data @%#x", e.Name, e.Ofs)
}
// ECreate is a file/store create error.
type ECreate struct {
Name string
Err error
}
func (e *ECreate) Error() string {
return fmt.Sprintf("%s: %s", e.Name, e.Err)
}
// EFreeList is a file/store format error.
type EFreeList struct {
Name string
Size int64
Block int64
}
func (e *EFreeList) Error() string {
return fmt.Sprintf("%s: invalid free list item, size %#x, block %#x", e.Name, e.Size, e.Block)
}
// EHandle is an error type reported for invalid Handles.
type EHandle struct {
Name string
Handle Handle
}
func (e EHandle) Error() string {
return fmt.Sprintf("%s: invalid handle %#x", e.Name, e.Handle)
}
// EHeader is a file/store format error.
type EHeader struct {
Name string
Header []byte
Expected []byte
}
func (e *EHeader) Error() string {
return fmt.Sprintf("%s: invalid header, got [% x], expected [% x]", e.Name, e.Header, e.Expected)
}
// ENullHandle is a file/store access error via a null handle.
type ENullHandle string
func (e ENullHandle) Error() string {
return fmt.Sprintf("%s: access via null handle", e)
}
// EOpen is a file/store open error.
type EOpen struct {
Name string
Err error
}
func (e *EOpen) Error() string {
return fmt.Sprintf("%s: %s", e.Name, e.Err)
}
// ERead is a file/store read error.
type ERead struct {
Name string
Ofs int64
Err error
}
func (e *ERead) Error() string {
return fmt.Sprintf("%s, %#x: %s", e.Name, e.Ofs, e.Err)
}
// ESize is a file/store size error.
type ESize struct {
Name string
Size int64
}
func (e *ESize) Error() string {
return fmt.Sprintf("%s: invalid size %#x(%d), size %%16 != 0", e.Name, e.Size, e.Size)
}
// EWrite is a file/store write error.
type EWrite struct {
Name string
Ofs int64
Err error
}
func (e *EWrite) Error() string {
return fmt.Sprintf("%s, %#x: %s", e.Name, e.Ofs, e.Err)
}

676
vendor/github.com/cznic/fileutil/falloc/falloc.go generated vendored Normal file
View File

@@ -0,0 +1,676 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
/*
This is an mostly (WIP) conforming implementation of the "specs" in docs.go.
The main incompletness is support for only one kind of FTL, though this table kind is still per "specs".
*/
package falloc
import (
"bytes"
"github.com/cznic/fileutil/storage"
"sync"
)
// Handle is a reference to a block in a file/store.
// Handle is an uint56 wrapped in an in64, i.e. the most significant byte must be always zero.
type Handle int64
// Put puts the 7 least significant bytes of h into b. The MSB of h should be zero.
func (h Handle) Put(b []byte) {
for ofs := 6; ofs >= 0; ofs-- {
b[ofs] = byte(h)
h >>= 8
}
}
// Get gets the 7 least significant bytes of h from b. The MSB of h is zeroed.
func (h *Handle) Get(b []byte) {
var x Handle
for ofs := 0; ofs <= 6; ofs++ {
x = x<<8 | Handle(b[ofs])
}
*h = x
}
// File is a file/store with space allocation/deallocation support.
type File struct {
f storage.Accessor
atoms int64 // current file size in atom units
canfree int64 // only blocks >= canfree can be subject to Free()
freetab [3857]int64 // freetab[0] is unused, freetab[1] is size 1 ptr, freetab[2] is size 2 ptr, ...
rwm sync.RWMutex
}
func (f *File) read(b []byte, off int64) {
if n, err := f.f.ReadAt(b, off); n != len(b) {
panic(&ERead{f.f.Name(), off, err})
}
}
func (f *File) write(b []byte, off int64) {
if n, err := f.f.WriteAt(b, off); n != len(b) {
panic(&EWrite{f.f.Name(), off, err})
}
}
var ( // R/O
hdr = []byte{0x0f, 0xf1, 0xc1, 0xa1, 0xfe, 0xa5, 0x1b, 0x1e, 0, 0, 0, 0, 0, 0, 2, 0} // free lists table @2
empty = make([]byte, 16)
zero = []byte{0}
zero7 = make([]byte, 7)
)
// New returns a new File backed by store or an error if any.
// Any existing data in store are discarded.
func New(store storage.Accessor) (f *File, err error) {
f = &File{f: store}
return f, storage.Mutate(store, func() (err error) {
if err = f.f.Truncate(0); err != nil {
return &ECreate{f.f.Name(), err}
}
if _, err = f.Alloc(hdr[1:]); err != nil { //TODO internal panicking versions of the exported fns.
return
}
if _, err = f.Alloc(nil); err != nil { // (empty) root @1
return
}
b := make([]byte, 3856*14)
for i := 1; i <= 3856; i++ {
Handle(i).Put(b[(i-1)*14:])
}
if _, err = f.Alloc(b); err != nil {
return
}
f.canfree = f.atoms
return
})
}
// Open returns a new File backed by store or an error if any.
// Store already has to be in a valid format.
func Open(store storage.Accessor) (f *File, err error) {
defer func() {
if e := recover(); e != nil {
f = nil
err = e.(error)
}
}()
fi, err := store.Stat()
if err != nil {
panic(&EOpen{store.Name(), err})
}
fs := fi.Size()
if fs&0xf != 0 {
panic(&ESize{store.Name(), fi.Size()})
}
f = &File{f: store, atoms: fs >> 4}
b := make([]byte, len(hdr))
f.read(b, 0)
if !bytes.Equal(b, hdr) {
panic(&EHeader{store.Name(), b, append([]byte{}, hdr...)})
}
var atoms int64
b, atoms = f.readUsed(2)
f.canfree = atoms + 2
ofs := 0
var size, p Handle
for ofs < len(b) {
size.Get(b[ofs:])
ofs += 7
p.Get(b[ofs:])
ofs += 7
if sz, pp := int64(size), int64(p); size == 0 || size > 3856 || (pp != 0 && pp < f.canfree) || pp<<4 > fs-16 {
panic(&EFreeList{store.Name(), sz, pp})
}
f.freetab[size] = int64(p)
}
return
}
// Accessor returns the File's underlying Accessor.
func (f *File) Accessor() storage.Accessor {
return f.f
}
// Close closes f and returns an error if any.
func (f *File) Close() (err error) {
return storage.Mutate(f.Accessor(), func() (err error) {
if err = f.f.Close(); err != nil {
err = &EClose{f.f.Name(), err}
}
return
})
}
// Root returns the handle of the DB root (top level directory, ...).
func (f *File) Root() Handle {
return 1
}
func (f *File) readUsed(atom int64) (content []byte, atoms int64) {
b, redirected := make([]byte, 7), false
redir:
ofs := atom << 4
f.read(b[:1], ofs)
switch pre := b[0]; {
default:
panic(&ECorrupted{f.f.Name(), ofs})
case pre == 0x00: // Empty block
case pre >= 1 && pre <= 237: // Short
content = make([]byte, pre)
f.read(content, ofs+1)
case pre >= 0xee && pre <= 0xfb: // Short esc
content = make([]byte, 15+16*(pre-0xee))
f.read(content, ofs+1)
content[len(content)-1] += 0xfe
case pre == 0xfc: // Long
f.read(b[:2], ofs+1)
n := int(b[0])<<8 + int(b[1])
switch {
default:
panic(&ECorrupted{f.f.Name(), ofs + 1})
case n >= 238 && n <= 61680: // Long non esc
content = make([]byte, n)
f.read(content, ofs+3)
case n >= 61681: // Long esc
content = make([]byte, 13+16*(n-0xf0f1))
f.read(content, ofs+3)
content[len(content)-1] += 0xfe
}
case pre == 0xfd: // redir
if redirected {
panic(&ECorrupted{f.f.Name(), ofs})
}
f.read(b[:7], ofs+1)
(*Handle)(&atom).Get(b)
redirected = true
goto redir
}
return content, rq2Atoms(len(content))
}
func (f *File) writeUsed(b []byte, atom int64) {
n := len(b)
switch ofs, atoms, endmark := atom<<4, rq2Atoms(n), true; {
default:
panic("internal error")
case n == 0:
f.write(empty, ofs)
case n <= 237:
if (n+1)&0xf == 0 { // content end == atom end
if v := b[n-1]; v >= 0xfe { // escape
pre := []byte{byte((16*0xee + n - 15) >> 4)}
f.write(pre, ofs)
f.write(b[:n-1], ofs+1)
f.write([]byte{v - 0xfe}, ofs+atoms<<4-1)
return
}
endmark = false
}
// non esacpe
pre := []byte{byte(n)}
f.write(pre, ofs)
f.write(b, ofs+1)
if endmark {
f.write(zero, ofs+atoms<<4-1) // last block byte <- used block
}
case n > 237 && n <= 61680:
if (n+3)&0xf == 0 { // content end == atom end
if v := b[n-1]; v >= 0xfe { // escape
x := (16*0xf0f1 + n - 13) >> 4
pre := []byte{0xFC, byte(x >> 8), byte(x)}
f.write(pre, ofs)
f.write(b[:n-1], ofs+3)
f.write([]byte{v - 0xfe}, ofs+atoms<<4-1)
return
}
endmark = false
}
// non esacpe
pre := []byte{0xfc, byte(n >> 8), byte(n)}
f.write(pre, ofs)
f.write(b, ofs+3)
if endmark {
f.write(zero, ofs+atoms<<4-1) // last block byte <- used block
}
}
}
func rq2Atoms(rqbytes int) (rqatoms int64) {
if rqbytes > 237 {
rqbytes += 2
}
return int64(rqbytes>>4 + 1)
}
func (f *File) extend(b []byte) (handle int64) {
handle = f.atoms
f.writeUsed(b, handle)
f.atoms += rq2Atoms(len(b))
return
}
// Alloc stores b in a newly allocated space and returns its handle and an error if any.
func (f *File) Alloc(b []byte) (handle Handle, err error) {
err = storage.Mutate(f.Accessor(), func() (err error) {
rqAtoms := rq2Atoms(len(b))
if rqAtoms > 3856 {
return &EBadRequest{f.f.Name(), len(b)}
}
for foundsize, foundp := range f.freetab[rqAtoms:] {
if foundp != 0 {
// this works only for the current unique sizes list (except the last item!)
size := int64(foundsize) + rqAtoms
handle = Handle(foundp)
if size == 3856 {
buf := make([]byte, 7)
f.read(buf, int64(handle)<<4+15)
(*Handle)(&size).Get(buf)
}
f.delFree(int64(handle), size)
if rqAtoms < size {
f.addFree(int64(handle)+rqAtoms, size-rqAtoms)
}
f.writeUsed(b, int64(handle))
return
}
}
handle = Handle(f.extend(b))
return
})
return
}
// checkLeft returns the atom size of a free bleck left adjacent to block @atom.
// If that block is not free the returned size is 0.
func (f *File) checkLeft(atom int64) (size int64) {
if atom <= f.canfree {
return
}
b := make([]byte, 7)
fp := atom << 4
f.read(b[:1], fp-1)
switch last := b[0]; {
case last <= 0xfd:
// used block
case last == 0xfe:
f.read(b, fp-8)
(*Handle)(&size).Get(b)
case last == 0xff:
size = 1
}
return
}
// getInfo returns the block @atom type and size.
func (f *File) getInfo(atom int64) (pref byte, size int64) {
b := make([]byte, 7)
fp := atom << 4
f.read(b[:1], fp)
switch pref = b[0]; {
case pref == 0: // Empty used
size = 1
case pref >= 1 && pref <= 237: // Short
size = rq2Atoms(int(pref))
case pref >= 0xee && pref <= 0xfb: // Short esc
size = rq2Atoms(15 + 16*int(pref-0xee))
case pref == 0xfc: // Long
f.read(b[:2], fp+1)
n := int(b[0])<<8 + int(b[1])
switch {
default:
panic(&ECorrupted{f.f.Name(), fp + 1})
case n >= 238 && n <= 61680: // Long non esc
size = rq2Atoms(n)
case n >= 61681: // Long esc
size = rq2Atoms(13 + 16*(n-0xf0f1))
}
case pref == 0xfd: // reloc
size = 1
case pref == 0xfe:
f.read(b, fp+15)
(*Handle)(&size).Get(b)
case pref == 0xff:
size = 1
}
return
}
// getSize returns the atom size of the block @atom and wheter it is free.
func (f *File) getSize(atom int64) (size int64, isFree bool) {
var typ byte
typ, size = f.getInfo(atom)
isFree = typ >= 0xfe
return
}
// checkRight returns the atom size of a free bleck right adjacent to block @atom,atoms.
// If that block is not free the returned size is 0.
func (f *File) checkRight(atom, atoms int64) (size int64) {
if atom+atoms >= f.atoms {
return
}
if sz, free := f.getSize(atom + atoms); free {
size = sz
}
return
}
// delFree removes the atoms@atom free block from the free block list
func (f *File) delFree(atom, atoms int64) {
b := make([]byte, 15)
size := int(atoms)
if n := len(f.freetab); atoms >= int64(n) {
size = n - 1
}
fp := atom << 4
f.read(b[1:], fp+1)
var prev, next Handle
prev.Get(b[1:])
next.Get(b[8:])
switch {
case prev == 0 && next != 0:
next.Put(b)
f.write(b[:7], int64(32+3+7+(size-1)*14))
f.write(zero7, int64(next)<<4+1)
f.freetab[size] = int64(next)
case prev != 0 && next == 0:
f.write(zero7, int64(prev)<<4+8)
case prev != 0 && next != 0:
prev.Put(b)
f.write(b[:7], int64(next)<<4+1)
next.Put(b)
f.write(b[:7], int64(prev)<<4+8)
default: // prev == 0 && next == 0:
f.write(zero7, int64(32+3+7+(size-1)*14))
f.freetab[size] = 0
}
}
// addFree adds atoms@atom to the free block lists and marks it free.
func (f *File) addFree(atom, atoms int64) {
b := make([]byte, 7)
size := int(atoms)
if n := len(f.freetab); atoms >= int64(n) {
size = n - 1
}
head := f.freetab[size]
if head == 0 { // empty list
f.makeFree(0, atom, atoms, 0)
Handle(atom).Put(b)
f.write(b, int64(32+3+7+(size-1)*14))
f.freetab[size] = atom
return
}
Handle(atom).Put(b)
f.write(b, head<<4+1) // head.prev = atom
f.makeFree(0, atom, atoms, head) // atom.next = head
f.write(b, int64(32+3+7+(size-1)*14))
f.freetab[size] = atom
}
// makeFree sets up the content of a free block atoms@atom, fills the prev and next links.
func (f *File) makeFree(prev, atom, atoms, next int64) {
b := make([]byte, 23)
fp := atom << 4
if atoms == 1 {
b[0] = 0xff
Handle(prev).Put(b[1:])
Handle(next).Put(b[8:])
b[15] = 0xff
f.write(b[:16], fp)
return
}
b[0] = 0xfe
Handle(prev).Put(b[1:])
Handle(next).Put(b[8:])
Handle(atoms).Put(b[15:])
f.write(b[:22], fp)
b[22] = 0xfe
f.write(b[15:], fp+atoms<<4-8)
}
// Read reads and return the data associated with handle and an error if any.
// Passing an invalid handle to Read may return invalid data without error.
// It's like getting garbage via passing an invalid pointer to C.memcopy().
func (f *File) Read(handle Handle) (b []byte, err error) {
defer func() {
if e := recover(); e != nil {
b = nil
err = e.(error)
}
}()
switch handle {
case 0:
panic(ENullHandle(f.f.Name()))
case 2:
panic(&EHandle{f.f.Name(), handle})
default:
b, _ = f.readUsed(int64(handle))
}
return
}
// Free frees space associated with handle and returns an error if any. Passing an invalid
// handle to Free or reusing handle afterwards will probably corrupt the database or provide
// invalid data on Read. It's like corrupting memory via passing an invalid pointer to C.free()
// or reusing that pointer.
func (f *File) Free(handle Handle) (err error) {
return storage.Mutate(f.Accessor(), func() (err error) {
atom := int64(handle)
atoms, isFree := f.getSize(atom)
if isFree || atom < f.canfree {
return &EHandle{f.f.Name(), handle}
}
leftFree, rightFree := f.checkLeft(atom), f.checkRight(atom, atoms)
switch {
case leftFree != 0 && rightFree != 0:
f.delFree(atom-leftFree, leftFree)
f.delFree(atom+atoms, rightFree)
f.addFree(atom-leftFree, leftFree+atoms+rightFree)
case leftFree != 0 && rightFree == 0:
f.delFree(atom-leftFree, leftFree)
if atom+atoms == f.atoms { // the left free neighbour and this block together are an empy tail
f.atoms = atom - leftFree
f.f.Truncate(f.atoms << 4)
return
}
f.addFree(atom-leftFree, leftFree+atoms)
case leftFree == 0 && rightFree != 0:
f.delFree(atom+atoms, rightFree)
f.addFree(atom, atoms+rightFree)
default: // leftFree == 0 && rightFree == 0
if atom+atoms < f.atoms { // isolated inner block
f.addFree(atom, atoms)
return
}
f.f.Truncate(atom << 4) // isolated tail block, shrink file
f.atoms = atom
}
return
})
}
// Realloc reallocates space associted with handle to acomodate b, returns the newhandle
// newly associated with b and an error if any. If keepHandle == true then Realloc guarantees
// newhandle == handle even if the new data are larger then the previous content associated
// with handle. If !keepHandle && newhandle != handle then reusing handle will probably corrupt
// the database.
// The above effects are like corrupting memory/data via passing an invalid pointer to C.realloc().
func (f *File) Realloc(handle Handle, b []byte, keepHandle bool) (newhandle Handle, err error) {
err = storage.Mutate(f.Accessor(), func() (err error) {
switch handle {
case 0, 2:
return &EHandle{f.f.Name(), handle}
case 1:
keepHandle = true
}
newhandle = handle
atom, newatoms := int64(handle), rq2Atoms(len(b))
if newatoms > 3856 {
return &EBadRequest{f.f.Name(), len(b)}
}
typ, oldatoms := f.getInfo(atom)
switch {
default:
return &ECorrupted{f.f.Name(), atom << 4}
case typ <= 0xfc: // non relocated used block
switch {
case newatoms == oldatoms: // in place replace
f.writeUsed(b, atom)
case newatoms < oldatoms: // in place shrink
rightFree := f.checkRight(atom, oldatoms)
if rightFree > 0 { // right join
f.delFree(atom+oldatoms, rightFree)
}
f.addFree(atom+newatoms, oldatoms+rightFree-newatoms)
f.writeUsed(b, atom)
case newatoms > oldatoms:
if rightFree := f.checkRight(atom, oldatoms); rightFree > 0 && newatoms <= oldatoms+rightFree {
f.delFree(atom+oldatoms, rightFree)
if newatoms < oldatoms+rightFree {
f.addFree(atom+newatoms, oldatoms+rightFree-newatoms)
}
f.writeUsed(b, atom)
return
}
if !keepHandle {
f.Free(Handle(atom))
newhandle, err = f.Alloc(b)
return
}
// reloc
newatom, e := f.Alloc(b)
if e != nil {
return e
}
buf := make([]byte, 16)
buf[0] = 0xfd
Handle(newatom).Put(buf[1:])
f.Realloc(Handle(atom), buf[1:], true)
f.write(buf[:1], atom<<4)
}
case typ == 0xfd: // reloc
var target Handle
buf := make([]byte, 7)
f.read(buf, atom<<4+1)
target.Get(buf)
switch {
case newatoms == 1:
f.writeUsed(b, atom)
f.Free(target)
default:
if rightFree := f.checkRight(atom, 1); rightFree > 0 && newatoms <= 1+rightFree {
f.delFree(atom+1, rightFree)
if newatoms < 1+rightFree {
f.addFree(atom+newatoms, 1+rightFree-newatoms)
}
f.writeUsed(b, atom)
f.Free(target)
return
}
newtarget, e := f.Realloc(Handle(target), b, false)
if e != nil {
return e
}
if newtarget != target {
Handle(newtarget).Put(buf)
f.write(buf, atom<<4+1)
}
}
}
return
})
return
}
// Lock locks f for writing. If the lock is already locked for reading or writing,
// Lock blocks until the lock is available. To ensure that the lock eventually becomes available,
// a blocked Lock call excludes new readers from acquiring the lock.
func (f *File) Lock() {
f.rwm.Lock()
}
// RLock locks f for reading. If the lock is already locked for writing or there is a writer
// already waiting to release the lock, RLock blocks until the writer has released the lock.
func (f *File) RLock() {
f.rwm.RLock()
}
// Unlock unlocks f for writing. It is a run-time error if f is not locked for writing on entry to Unlock.
//
// As with Mutexes, a locked RWMutex is not associated with a particular goroutine.
// One goroutine may RLock (Lock) f and then arrange for another goroutine to RUnlock (Unlock) it.
func (f *File) Unlock() {
f.rwm.Unlock()
}
// RUnlock undoes a single RLock call; it does not affect other simultaneous readers.
// It is a run-time error if f is not locked for reading on entry to RUnlock.
func (f *File) RUnlock() {
f.rwm.RUnlock()
}
// LockedAlloc wraps Alloc in a Lock/Unlock pair.
func (f *File) LockedAlloc(b []byte) (handle Handle, err error) {
f.Lock()
defer f.Unlock()
return f.Alloc(b)
}
// LockedFree wraps Free in a Lock/Unlock pair.
func (f *File) LockedFree(handle Handle) (err error) {
f.Lock()
defer f.Unlock()
return f.Free(handle)
}
// LockedRead wraps Read in a RLock/RUnlock pair.
func (f *File) LockedRead(handle Handle) (b []byte, err error) {
f.RLock()
defer f.RUnlock()
return f.Read(handle)
}
// LockedRealloc wraps Realloc in a Lock/Unlock pair.
func (f *File) LockedRealloc(handle Handle, b []byte, keepHandle bool) (newhandle Handle, err error) {
f.Lock()
defer f.Unlock()
return f.Realloc(handle, b, keepHandle)
}

15
vendor/github.com/cznic/fileutil/falloc/test_deps.go generated vendored Normal file
View File

@@ -0,0 +1,15 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package falloc
// Pull test dependencies too.
// Enables easy 'go test X' after 'go get X'
import (
_ "github.com/cznic/fileutil"
_ "github.com/cznic/fileutil/storage"
_ "github.com/cznic/mathutil"
)

223
vendor/github.com/cznic/fileutil/fileutil.go generated vendored Normal file
View File

@@ -0,0 +1,223 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package fileutil collects some file utility functions.
package fileutil
import (
"fmt"
"io"
"os"
"path/filepath"
"runtime"
"strconv"
"sync"
"time"
)
// GoMFile is a concurrent access safe version of MFile.
type GoMFile struct {
mfile *MFile
mutex sync.Mutex
}
// NewGoMFile return a newly created GoMFile.
func NewGoMFile(fname string, flag int, perm os.FileMode, delta_ns int64) (m *GoMFile, err error) {
m = &GoMFile{}
if m.mfile, err = NewMFile(fname, flag, perm, delta_ns); err != nil {
m = nil
}
return
}
func (m *GoMFile) File() (file *os.File, err error) {
m.mutex.Lock()
defer m.mutex.Unlock()
return m.mfile.File()
}
func (m *GoMFile) SetChanged() {
m.mutex.Lock()
defer m.mutex.Unlock()
m.mfile.SetChanged()
}
func (m *GoMFile) SetHandler(h MFileHandler) {
m.mutex.Lock()
defer m.mutex.Unlock()
m.mfile.SetHandler(h)
}
// MFileHandler resolves modifications of File.
// Possible File context is expected to be a part of the handler's closure.
type MFileHandler func(*os.File) error
// MFile represents an os.File with a guard/handler on change/modification.
// Example use case is an app with a configuration file which can be modified at any time
// and have to be reloaded in such event prior to performing something configurable by that
// file. The checks are made only on access to the MFile file by
// File() and a time threshold/hysteresis value can be chosen on creating a new MFile.
type MFile struct {
file *os.File
handler MFileHandler
t0 int64
delta int64
ctime int64
}
// NewMFile returns a newly created MFile or Error if any.
// The fname, flag and perm parameters have the same meaning as in os.Open.
// For meaning of the delta_ns parameter please see the (m *MFile) File() docs.
func NewMFile(fname string, flag int, perm os.FileMode, delta_ns int64) (m *MFile, err error) {
m = &MFile{}
m.t0 = time.Now().UnixNano()
if m.file, err = os.OpenFile(fname, flag, perm); err != nil {
return
}
var fi os.FileInfo
if fi, err = m.file.Stat(); err != nil {
return
}
m.ctime = fi.ModTime().UnixNano()
m.delta = delta_ns
runtime.SetFinalizer(m, func(m *MFile) {
m.file.Close()
})
return
}
// SetChanged forces next File() to unconditionally handle modification of the wrapped os.File.
func (m *MFile) SetChanged() {
m.ctime = -1
}
// SetHandler sets a function to be invoked when modification of MFile is to be processed.
func (m *MFile) SetHandler(h MFileHandler) {
m.handler = h
}
// File returns an os.File from MFile. If time elapsed between the last invocation of this function
// and now is at least delta_ns ns (a parameter of NewMFile) then the file is checked for
// change/modification. For delta_ns == 0 the modification is checked w/o getting os.Time().
// If a change is detected a handler is invoked on the MFile file.
// Any of these steps can produce an Error. If that happens the function returns nil, Error.
func (m *MFile) File() (file *os.File, err error) {
var now int64
mustCheck := m.delta == 0
if !mustCheck {
now = time.Now().UnixNano()
mustCheck = now-m.t0 > m.delta
}
if mustCheck { // check interval reached
var fi os.FileInfo
if fi, err = m.file.Stat(); err != nil {
return
}
if fi.ModTime().UnixNano() != m.ctime { // modification detected
if m.handler == nil {
return nil, fmt.Errorf("no handler set for modified file %q", m.file.Name())
}
if err = m.handler(m.file); err != nil {
return
}
m.ctime = fi.ModTime().UnixNano()
}
m.t0 = now
}
return m.file, nil
}
// Read reads buf from r. It will either fill the full buf or fail.
// It wraps the functionality of an io.Reader which may return less bytes than requested,
// but may block if not all data are ready for the io.Reader.
func Read(r io.Reader, buf []byte) (err error) {
have := 0
remain := len(buf)
got := 0
for remain > 0 {
if got, err = r.Read(buf[have:]); err != nil {
return
}
remain -= got
have += got
}
return
}
// "os" and/or "syscall" extensions
// FadviseAdvice is used by Fadvise.
type FadviseAdvice int
// FAdviseAdvice values.
const (
// $ grep FADV /usr/include/bits/fcntl.h
POSIX_FADV_NORMAL FadviseAdvice = iota // No further special treatment.
POSIX_FADV_RANDOM // Expect random page references.
POSIX_FADV_SEQUENTIAL // Expect sequential page references.
POSIX_FADV_WILLNEED // Will need these pages.
POSIX_FADV_DONTNEED // Don't need these pages.
POSIX_FADV_NOREUSE // Data will be accessed once.
)
// TempFile creates a new temporary file in the directory dir with a name
// ending with suffix, basename starting with prefix, opens the file for
// reading and writing, and returns the resulting *os.File. If dir is the
// empty string, TempFile uses the default directory for temporary files (see
// os.TempDir). Multiple programs calling TempFile simultaneously will not
// choose the same file. The caller can use f.Name() to find the pathname of
// the file. It is the caller's responsibility to remove the file when no
// longer needed.
//
// NOTE: This function differs from ioutil.TempFile.
func TempFile(dir, prefix, suffix string) (f *os.File, err error) {
if dir == "" {
dir = os.TempDir()
}
nconflict := 0
for i := 0; i < 10000; i++ {
name := filepath.Join(dir, prefix+nextInfix()+suffix)
f, err = os.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0600)
if os.IsExist(err) {
if nconflict++; nconflict > 10 {
rand = reseed()
}
continue
}
break
}
return
}
// Random number state.
// We generate random temporary file names so that there's a good
// chance the file doesn't exist yet - keeps the number of tries in
// TempFile to a minimum.
var rand uint32
var randmu sync.Mutex
func reseed() uint32 {
return uint32(time.Now().UnixNano() + int64(os.Getpid()))
}
func nextInfix() string {
randmu.Lock()
r := rand
if r == 0 {
r = reseed()
}
r = r*1664525 + 1013904223 // constants from Numerical Recipes
rand = r
randmu.Unlock()
return strconv.Itoa(int(1e9 + r%1e9))[1:]
}

25
vendor/github.com/cznic/fileutil/fileutil_arm.go generated vendored Normal file
View File

@@ -0,0 +1,25 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"io"
"os"
)
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Not supported on ARM.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Not supported on ARM.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

25
vendor/github.com/cznic/fileutil/fileutil_darwin.go generated vendored Normal file
View File

@@ -0,0 +1,25 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"io"
"os"
)
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Not supported on OSX.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Not supported on OSX.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

27
vendor/github.com/cznic/fileutil/fileutil_freebsd.go generated vendored Normal file
View File

@@ -0,0 +1,27 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build !arm
package fileutil
import (
"io"
"os"
)
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Unimplemented on FreeBSD.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Unimplemented on FreeBSD.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

96
vendor/github.com/cznic/fileutil/fileutil_linux.go generated vendored Normal file
View File

@@ -0,0 +1,96 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build !arm
package fileutil
import (
"bytes"
"io"
"io/ioutil"
"os"
"strconv"
"syscall"
)
func n(s []byte) byte {
for i, c := range s {
if c < '0' || c > '9' {
s = s[:i]
break
}
}
v, _ := strconv.Atoi(string(s))
return byte(v)
}
func init() {
b, err := ioutil.ReadFile("/proc/sys/kernel/osrelease")
if err != nil {
panic(err)
}
tokens := bytes.Split(b, []byte("."))
if len(tokens) > 3 {
tokens = tokens[:3]
}
switch len(tokens) {
case 3:
// Supported since kernel 2.6.38
if bytes.Compare([]byte{n(tokens[0]), n(tokens[1]), n(tokens[2])}, []byte{2, 6, 38}) < 0 {
puncher = func(*os.File, int64, int64) error { return nil }
}
case 2:
if bytes.Compare([]byte{n(tokens[0]), n(tokens[1])}, []byte{2, 7}) < 0 {
puncher = func(*os.File, int64, int64) error { return nil }
}
default:
puncher = func(*os.File, int64, int64) error { return nil }
}
}
var puncher = func(f *os.File, off, len int64) error {
const (
/*
/usr/include/linux$ grep FL_ falloc.h
*/
_FALLOC_FL_KEEP_SIZE = 0x01 // default is extend size
_FALLOC_FL_PUNCH_HOLE = 0x02 // de-allocates range
)
_, _, errno := syscall.Syscall6(
syscall.SYS_FALLOCATE,
uintptr(f.Fd()),
uintptr(_FALLOC_FL_KEEP_SIZE|_FALLOC_FL_PUNCH_HOLE),
uintptr(off),
uintptr(len),
0, 0)
if errno != 0 {
return os.NewSyscallError("SYS_FALLOCATE", errno)
}
return nil
}
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. No-op for kernels < 2.6.38 (or < 2.7).
func PunchHole(f *os.File, off, len int64) error {
return puncher(f, off, len)
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
_, _, errno := syscall.Syscall6(
syscall.SYS_FADVISE64,
uintptr(f.Fd()),
uintptr(off),
uintptr(len),
uintptr(advice),
0, 0)
return os.NewSyscallError("SYS_FADVISE64", errno)
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

27
vendor/github.com/cznic/fileutil/fileutil_netbsd.go generated vendored Normal file
View File

@@ -0,0 +1,27 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build !arm
package fileutil
import (
"io"
"os"
)
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Similar to FreeBSD, this is
// unimplemented.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Unimplemented on NetBSD.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

25
vendor/github.com/cznic/fileutil/fileutil_openbsd.go generated vendored Normal file
View File

@@ -0,0 +1,25 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"io"
"os"
)
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Similar to FreeBSD, this is
// unimplemented.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Unimplemented on OpenBSD.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

25
vendor/github.com/cznic/fileutil/fileutil_plan9.go generated vendored Normal file
View File

@@ -0,0 +1,25 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"io"
"os"
)
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Unimplemented on Plan 9.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Unimplemented on Plan 9.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

27
vendor/github.com/cznic/fileutil/fileutil_solaris.go generated vendored Normal file
View File

@@ -0,0 +1,27 @@
// Copyright (c) 2013 jnml. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build go1.3
package fileutil
import (
"io"
"os"
)
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Not supported on Solaris.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Not supported on Solaris.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

183
vendor/github.com/cznic/fileutil/fileutil_windows.go generated vendored Normal file
View File

@@ -0,0 +1,183 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"io"
"os"
"sync"
"syscall"
"unsafe"
)
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Not supported on Windows.
func PunchHole(f *os.File, off, len int64) error {
return puncher(f, off, len)
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Not supported on Windows.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool {
if err == io.EOF {
return true
}
// http://social.technet.microsoft.com/Forums/windowsserver/en-US/1a16311b-c625-46cf-830b-6a26af488435/how-to-solve-error-38-0x26-errorhandleeof-using-fsctlgetretrievalpointers
x, ok := err.(*os.PathError)
return ok && x.Op == "read" && x.Err.(syscall.Errno) == 0x26
}
var (
modkernel32 = syscall.NewLazyDLL("kernel32.dll")
procDeviceIOControl = modkernel32.NewProc("DeviceIoControl")
sparseFilesMu sync.Mutex
sparseFiles map[uintptr]struct{}
)
func init() {
// sparseFiles is an fd set for already "sparsed" files - according to
// msdn.microsoft.com/en-us/library/windows/desktop/aa364225(v=vs.85).aspx
// the file handles are unique per process.
sparseFiles = make(map[uintptr]struct{})
}
// puncHoleWindows punches a hole into the given file starting at offset,
// measuring "size" bytes
// (http://msdn.microsoft.com/en-us/library/windows/desktop/aa364597%28v=vs.85%29.aspx)
func puncher(file *os.File, offset, size int64) error {
if err := ensureFileSparse(file); err != nil {
return err
}
// http://msdn.microsoft.com/en-us/library/windows/desktop/aa364411%28v=vs.85%29.aspx
// typedef struct _FILE_ZERO_DATA_INFORMATION {
// LARGE_INTEGER FileOffset;
// LARGE_INTEGER BeyondFinalZero;
//} FILE_ZERO_DATA_INFORMATION, *PFILE_ZERO_DATA_INFORMATION;
type fileZeroDataInformation struct {
FileOffset, BeyondFinalZero int64
}
lpInBuffer := fileZeroDataInformation{
FileOffset: offset,
BeyondFinalZero: offset + size}
return deviceIOControl(false, file.Fd(), uintptr(unsafe.Pointer(&lpInBuffer)), 16)
}
// // http://msdn.microsoft.com/en-us/library/windows/desktop/cc948908%28v=vs.85%29.aspx
// type fileSetSparseBuffer struct {
// SetSparse bool
// }
func ensureFileSparse(file *os.File) (err error) {
fd := file.Fd()
sparseFilesMu.Lock()
if _, ok := sparseFiles[fd]; ok {
sparseFilesMu.Unlock()
return nil
}
if err = deviceIOControl(true, fd, 0, 0); err == nil {
sparseFiles[fd] = struct{}{}
}
sparseFilesMu.Unlock()
return err
}
func deviceIOControl(setSparse bool, fd, inBuf, inBufLen uintptr) (err error) {
const (
//http://source.winehq.org/source/include/winnt.h#L4605
file_read_data = 1
file_write_data = 2
// METHOD_BUFFERED 0
method_buffered = 0
// FILE_ANY_ACCESS 0
file_any_access = 0
// FILE_DEVICE_FILE_SYSTEM 0x00000009
file_device_file_system = 0x00000009
// FILE_SPECIAL_ACCESS (FILE_ANY_ACCESS)
file_special_access = file_any_access
file_read_access = file_read_data
file_write_access = file_write_data
// http://source.winehq.org/source/include/winioctl.h
// #define CTL_CODE ( DeviceType,
// Function,
// Method,
// Access )
// ((DeviceType) << 16) | ((Access) << 14) | ((Function) << 2) | (Method)
// FSCTL_SET_COMPRESSION CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 16, METHOD_BUFFERED, FILE_READ_DATA | FILE_WRITE_DATA)
fsctl_set_compression = (file_device_file_system << 16) | ((file_read_access | file_write_access) << 14) | (16 << 2) | method_buffered
// FSCTL_SET_SPARSE CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 49, METHOD_BUFFERED, FILE_SPECIAL_ACCESS)
fsctl_set_sparse = (file_device_file_system << 16) | (file_special_access << 14) | (49 << 2) | method_buffered
// FSCTL_SET_ZERO_DATA CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 50, METHOD_BUFFERED, FILE_WRITE_DATA)
fsctl_set_zero_data = (file_device_file_system << 16) | (file_write_data << 14) | (50 << 2) | method_buffered
)
retPtr := uintptr(unsafe.Pointer(&(make([]byte, 8)[0])))
var r1 uintptr
var e1 syscall.Errno
if setSparse {
// BOOL
// WINAPI
// DeviceIoControl( (HANDLE) hDevice, // handle to a file
// FSCTL_SET_SPARSE, // dwIoControlCode
// (PFILE_SET_SPARSE_BUFFER) lpInBuffer, // input buffer
// (DWORD) nInBufferSize, // size of input buffer
// NULL, // lpOutBuffer
// 0, // nOutBufferSize
// (LPDWORD) lpBytesReturned, // number of bytes returned
// (LPOVERLAPPED) lpOverlapped ); // OVERLAPPED structure
r1, _, e1 = syscall.Syscall9(procDeviceIOControl.Addr(), 8,
fd,
uintptr(fsctl_set_sparse),
// If the lpInBuffer parameter is NULL, the operation will behave the same as if the SetSparse member of the FILE_SET_SPARSE_BUFFER structure were TRUE. In other words, the operation sets the file to a sparse file.
0, // uintptr(unsafe.Pointer(&lpInBuffer)),
0, // 1,
0,
0,
retPtr,
0,
0)
} else {
// BOOL
// WINAPI
// DeviceIoControl( (HANDLE) hDevice, // handle to a file
// FSCTL_SET_ZERO_DATA, // dwIoControlCode
// (LPVOID) lpInBuffer, // input buffer
// (DWORD) nInBufferSize, // size of input buffer
// NULL, // lpOutBuffer
// 0, // nOutBufferSize
// (LPDWORD) lpBytesReturned, // number of bytes returned
// (LPOVERLAPPED) lpOverlapped ); // OVERLAPPED structure
r1, _, e1 = syscall.Syscall9(procDeviceIOControl.Addr(), 8,
fd,
uintptr(fsctl_set_zero_data),
inBuf,
inBufLen,
0,
0,
retPtr,
0,
0)
}
if r1 == 0 {
if e1 != 0 {
err = error(e1)
} else {
err = syscall.EINVAL
}
}
return err
}

153
vendor/github.com/cznic/fileutil/hdb/hdb.go generated vendored Normal file
View File

@@ -0,0 +1,153 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
/*
WIP: Package hdb provides a "handle"/value DB like store, but actually it's
closer to the model of a process's virtual memory and its alloc, free and move
methods.
The hdb package is a thin layer around falloc.File providing stable-only
handles and the basic synchronizing primitives. The central functionality of
hdb are the New, Set, Get and Delete methods of Store.
Conceptual analogy:
New alloc(sizeof(content)), return new "memory" pointer (a handle).
Get memmove() from "memory" "pointed to" by handle to the result content.
Note: Handle "knows" the size of its content.
Set memmove() from content to "memory" pointed to by handle.
In contrast to real memory, the new content may have different
size than the previously stored one w/o additional handling
and the "pointer" handle remains the same.
Delete free() the "memory" "pointed to" by handle.
*/
package hdb
import (
"github.com/cznic/fileutil/falloc"
"github.com/cznic/fileutil/storage"
)
type Store struct {
f *falloc.File
}
// New returns a newly created Store backed by accessor, discarding its conents if any.
// If successful, methods on the returned Store can be used for I/O.
// It returns the Store and an error, if any.
func New(accessor storage.Accessor) (store *Store, err error) {
s := &Store{}
if s.f, err = falloc.New(accessor); err == nil {
store = s
}
return
}
// Open opens the Store from accessor.
// If successful, methods on the returned Store can be used for data exchange.
// It returns the Store and an error, if any.
func Open(accessor storage.Accessor) (store *Store, err error) {
s := &Store{}
if s.f, err = falloc.Open(accessor); err == nil {
store = s
}
return
}
// Close closes the store. Further access to the store has undefined behavior and may panic.
// It returns an error, if any.
func (s *Store) Close() (err error) {
defer func() {
s.f = nil
}()
return s.f.Close()
}
// Delete deletes the data associated with handle.
// It returns an error if any.
func (s *Store) Delete(handle falloc.Handle) (err error) {
return s.f.Free(handle)
}
// Get gets the data associated with handle.
// It returns the data and an error, if any.
func (s *Store) Get(handle falloc.Handle) (b []byte, err error) {
return s.f.Read(handle)
}
// New associates data with a new handle.
// It returns the handle and an error, if any.
func (s *Store) New(b []byte) (handle falloc.Handle, err error) {
return s.f.Alloc(b)
}
// Set associates data with an existing handle.
// It returns an error, if any.
func (s *Store) Set(handle falloc.Handle, b []byte) (err error) {
_, err = s.f.Realloc(handle, b, true)
return
}
// Root returns the handle of the DB root (top level directory, ...).
func (s *Store) Root() falloc.Handle {
return s.f.Root()
}
// File returns the underlying falloc.File of 's'.
func (s *Store) File() *falloc.File {
return s.f
}
// Lock locks 's' for writing. If the lock is already locked for reading or writing,
// Lock blocks until the lock is available. To ensure that the lock eventually becomes available,
// a blocked Lock call excludes new readers from acquiring the lock.
func (s *Store) Lock() {
s.f.Lock()
}
// RLock locks 's' for reading. If the lock is already locked for writing or there is a writer
// already waiting to release the lock, RLock blocks until the writer has released the lock.
func (s *Store) RLock() {
s.f.RLock()
}
// Unlock unlocks 's' for writing. It's a run-time error if 's' is not locked for writing on entry to Unlock.
//
// As with Mutexes, a locked RWMutex is not associated with a particular goroutine.
// One goroutine may RLock (Lock) 's' and then arrange for another goroutine to RUnlock (Unlock) it.
func (s *Store) Unlock() {
s.f.Unlock()
}
// RUnlock undoes a single RLock call; it does not affect other simultaneous readers.
// It's a run-time error if 's' is not locked for reading on entry to RUnlock.
func (s *Store) RUnlock() {
s.f.RUnlock()
}
// LockedNew wraps New in a Lock/Unlock pair.
func (s *Store) LockedNew(b []byte) (handle falloc.Handle, err error) {
return s.f.LockedAlloc(b)
}
// LockedDelete wraps Delete in a Lock/Unlock pair.
func (s *Store) LockedDelete(handle falloc.Handle) (err error) {
return s.f.LockedFree(handle)
}
// LockedGet wraps Get in a RLock/RUnlock pair.
func (s *Store) LockedGet(handle falloc.Handle) (b []byte, err error) {
return s.f.LockedRead(handle)
}
// LockedSet wraps Set in a Lock/Unlock pair.
func (s *Store) LockedSet(handle falloc.Handle, b []byte) (err error) {
_, err = s.f.Realloc(handle, b, true)
return
}

13
vendor/github.com/cznic/fileutil/hdb/test_deps.go generated vendored Normal file
View File

@@ -0,0 +1,13 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package hdb
// Pull test dependencies too.
// Enables easy 'go test X' after 'go get X'
import (
// nothing yet
)

322
vendor/github.com/cznic/fileutil/storage/cache.go generated vendored Normal file
View File

@@ -0,0 +1,322 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package storage
import (
"container/list"
"io"
"math"
"os"
"sync"
"sync/atomic"
)
type cachepage struct {
b [512]byte
dirty bool
lru *list.Element
pi int64
valid int // page content is b[:valid]
}
func (p *cachepage) wr(b []byte, off int) (wasDirty bool) {
copy(p.b[off:], b)
if n := off + len(b); n > p.valid {
p.valid = n
}
wasDirty = p.dirty
p.dirty = true
return
}
func (c *Cache) rd(off int64, read bool) (p *cachepage, ok bool) {
c.Rq++
pi := off >> 9
if p, ok = c.m[pi]; ok {
c.lru.MoveToBack(p.lru)
return
}
if !read {
return
}
fp := off &^ 511
if fp >= c.size {
return
}
rq := 512
if fp+512 > c.size {
rq = int(c.size - fp)
}
p = &cachepage{pi: pi, valid: rq}
p.lru = c.lru.PushBack(p)
if n, err := c.f.ReadAt(p.b[:p.valid], fp); n != rq {
panic(err)
}
c.Load++
if c.advise != nil {
c.advise(fp, 512, false)
}
c.m[pi], ok = p, true
return
}
func (c *Cache) wr(off int64) (p *cachepage) {
var ok bool
if p, ok = c.rd(off, false); ok {
return
}
pi := off >> 9
p = &cachepage{pi: pi}
p.lru = c.lru.PushBack(p)
c.m[pi] = p
return
}
// Cache provides caching support for another store Accessor.
type Cache struct {
advise func(int64, int, bool)
clean chan bool
cleaning int32
close chan bool
f Accessor
fi *FileInfo
lock sync.Mutex
lru *list.List
m map[int64]*cachepage
maxpages int
size int64
sync chan bool
wlist *list.List
write chan bool
writing int32
Rq int64 // Pages requested from cache
Load int64 // Pages loaded (cache miss)
Purge int64 // Pages purged
Top int // "High water" pages
}
// Implementation of Accessor.
func (c *Cache) BeginUpdate() error { return nil }
// Implementation of Accessor.
func (c *Cache) EndUpdate() error { return nil }
// NewCache creates a caching Accessor from store with total of maxcache bytes.
// NewCache returns the new Cache, implementing Accessor or an error if any.
//
// The LRU mechanism is used, so the cache tries to keep often accessed pages cached.
//
func NewCache(store Accessor, maxcache int64, advise func(int64, int, bool)) (c *Cache, err error) {
var fi os.FileInfo
if fi, err = store.Stat(); err != nil {
return
}
x := maxcache >> 9
if x > math.MaxInt32/2 {
x = math.MaxInt32 / 2
}
c = &Cache{
advise: advise,
clean: make(chan bool, 1),
close: make(chan bool),
f: store,
lru: list.New(), // front == oldest used, back == last recently used
m: make(map[int64]*cachepage),
maxpages: int(x),
size: fi.Size(),
sync: make(chan bool),
wlist: list.New(),
write: make(chan bool, 1),
}
c.fi = NewFileInfo(fi, c)
go c.writer()
go c.cleaner(int((int64(c.maxpages) * 95) / 100)) // hysteresis
return
}
func (c *Cache) Accessor() Accessor {
return c.f
}
func (c *Cache) Close() (err error) {
close(c.write)
<-c.close
close(c.clean)
<-c.close
return c.f.Close()
}
func (c *Cache) Name() (s string) {
return c.f.Name()
}
func (c *Cache) ReadAt(b []byte, off int64) (n int, err error) {
po := int(off) & 0x1ff
bp := 0
rem := len(b)
m := 0
for rem != 0 {
c.lock.Lock() // X1+
p, ok := c.rd(off, true)
if !ok {
c.lock.Unlock() // X1-
return -1, io.EOF
}
rq := rem
if po+rq > 512 {
rq = 512 - po
}
if n := copy(b[bp:bp+rq], p.b[po:p.valid]); n != rq {
c.lock.Unlock() // X1-
return -1, io.EOF
}
m = len(c.m)
c.lock.Unlock() // X1-
po = 0
bp += rq
off += int64(rq)
rem -= rq
n += rq
}
if m > c.maxpages && atomic.CompareAndSwapInt32(&c.cleaning, 0, 1) {
if m > c.Top {
c.Top = m
}
c.clean <- true
}
return
}
func (c *Cache) Stat() (fi os.FileInfo, err error) {
c.lock.Lock()
defer c.lock.Unlock()
return c.fi, nil
}
func (c *Cache) Sync() (err error) {
c.write <- false
<-c.sync
return
}
func (c *Cache) Truncate(size int64) (err error) {
c.Sync() //TODO improve (discard pages, the writer goroutine should also be aware, ...)
c.lock.Lock()
defer c.lock.Unlock()
c.size = size
return c.f.Truncate(size)
}
func (c *Cache) WriteAt(b []byte, off int64) (n int, err error) {
po := int(off) & 0x1ff
bp := 0
rem := len(b)
m := 0
for rem != 0 {
c.lock.Lock() // X+
p := c.wr(off)
rq := rem
if po+rq > 512 {
rq = 512 - po
}
if wasDirty := p.wr(b[bp:bp+rq], po); !wasDirty {
c.wlist.PushBack(p)
}
m = len(c.m)
po = 0
bp += rq
off += int64(rq)
if off > c.size {
c.size = off
}
c.lock.Unlock() // X-
rem -= rq
n += rq
}
if atomic.CompareAndSwapInt32(&c.writing, 0, 1) {
c.write <- true
}
if m > c.maxpages && atomic.CompareAndSwapInt32(&c.cleaning, 0, 1) {
if m > c.Top {
c.Top = m
}
c.clean <- true
}
return
}
func (c *Cache) writer() {
for ok := true; ok; {
var wr bool
var off int64
wr, ok = <-c.write
for {
c.lock.Lock() // X1+
item := c.wlist.Front()
if item == nil {
c.lock.Unlock() // X1-
break
}
p := item.Value.(*cachepage)
off = p.pi << 9
if n, err := c.f.WriteAt(p.b[:p.valid], off); n != p.valid {
c.lock.Unlock() // X1-
panic("TODO Cache.writer errchan") //TODO +errchan
panic(err)
}
p.dirty = false
c.wlist.Remove(item)
if c.advise != nil {
c.advise(off, 512, true)
}
c.lock.Unlock() // X1-
}
switch {
case wr:
atomic.AddInt32(&c.writing, -1)
case ok:
c.sync <- true
}
}
c.close <- true
}
func (c *Cache) cleaner(limit int) {
for _ = range c.clean {
var item *list.Element
for {
c.lock.Lock() // X1+
if len(c.m) < limit {
c.lock.Unlock() // X1-
break
}
if item == nil {
item = c.lru.Front()
}
if p := item.Value.(*cachepage); !p.dirty {
delete(c.m, p.pi)
c.lru.Remove(item)
c.Purge++
}
item = item.Next()
c.lock.Unlock() // X1-
}
atomic.AddInt32(&c.cleaning, -1)
}
c.close <- true
}

50
vendor/github.com/cznic/fileutil/storage/file.go generated vendored Normal file
View File

@@ -0,0 +1,50 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package storage
import (
"os"
)
// FileAccessor is the concrete type returned by NewFile and OpenFile.
type FileAccessor struct {
*os.File
}
// Implementation of Accessor.
func (f *FileAccessor) BeginUpdate() error { return nil }
// Implementation of Accessor.
func (f *FileAccessor) EndUpdate() error { return nil }
// NewFile returns an Accessor backed by an os.File named name, It opens the
// named file with specified flag (os.O_RDWR etc.) and perm, (0666 etc.) if
// applicable. If successful, methods on the returned Accessor can be used for
// I/O. It returns the Accessor and an Error, if any.
//
// NOTE: The returned Accessor implements BeginUpdate and EndUpdate as a no op.
func NewFile(name string, flag int, perm os.FileMode) (store Accessor, err error) {
var f FileAccessor
if f.File, err = os.OpenFile(name, flag, perm); err == nil {
store = &f
}
return
}
// OpenFile returns an Accessor backed by an existing os.File named name, It
// opens the named file with specified flag (os.O_RDWR etc.) and perm, (0666
// etc.) if applicable. If successful, methods on the returned Accessor can be
// used for I/O. It returns the Accessor and an Error, if any.
//
// NOTE: The returned Accessor implements BeginUpdate and EndUpdate as a no op.
func OpenFile(name string, flag int, perm os.FileMode) (store Accessor, err error) {
var f FileAccessor
if f.File, err = os.OpenFile(name, flag, perm); err == nil {
store = &f
}
return
}

161
vendor/github.com/cznic/fileutil/storage/mem.go generated vendored Normal file
View File

@@ -0,0 +1,161 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package storage
import (
"errors"
"fmt"
"io/ioutil"
"math"
"os"
)
//TODO -> exported type w/ exported fields
type memaccessor struct {
f *os.File
fi *FileInfo
b []byte
}
// Implementation of Accessor.
func (m *memaccessor) BeginUpdate() error { return nil }
// Implementation of Accessor.
func (f *memaccessor) EndUpdate() error { return nil }
// NewMem returns a new Accessor backed by an os.File. The returned Accessor
// keeps all of the store content in memory. The memory and file images are
// synced only by Sync and Close. Recomended for small amounts of data only
// and content which may be lost on process kill/crash. NewMem return the
// Accessor or an error of any.
//
// NOTE: The returned Accessor implements BeginUpdate and EndUpdate as a no op.
func NewMem(f *os.File) (store Accessor, err error) {
a := &memaccessor{f: f}
if err = f.Truncate(0); err != nil {
return
}
var fi os.FileInfo
if fi, err = a.f.Stat(); err != nil {
return
}
a.fi = NewFileInfo(fi, a)
store = a
return
}
// OpenMem return a new Accessor backed by an os.File. The store content is
// loaded from f. The returned Accessor keeps all of the store content in
// memory. The memory and file images are synced only Sync and Close.
// Recomended for small amounts of data only and content which may be lost on
// process kill/crash. OpenMem return the Accessor or an error of any.
//
// NOTE: The returned Accessor implements BeginUpdate and EndUpdate as a no op.
func OpenMem(f *os.File) (store Accessor, err error) {
a := &memaccessor{f: f}
if a.b, err = ioutil.ReadAll(a.f); err != nil {
a.f.Close()
return
}
var fi os.FileInfo
if fi, err = a.f.Stat(); err != nil {
a.f.Close()
return
}
a.fi = NewFileInfo(fi, a)
store = a
return
}
// Close implements Accessor. Specifically it synchronizes the memory and file images.
func (a *memaccessor) Close() (err error) {
defer func() {
a.b = nil
if a.f != nil {
if e := a.f.Close(); e != nil && err == nil {
err = e
}
}
a.f = nil
}()
return a.Sync()
}
func (a *memaccessor) Name() string {
return a.f.Name()
}
func (a *memaccessor) ReadAt(b []byte, off int64) (n int, err error) {
if off < 0 || off > math.MaxInt32 {
return -1, fmt.Errorf("ReadAt: illegal offset %#x", off)
}
rq, fp := len(b), int(off)
if fp+rq > len(a.b) {
return -1, fmt.Errorf("ReadAt: illegal rq %#x @ offset %#x, len %#x", rq, fp, len(a.b))
}
copy(b, a.b[fp:])
return
}
func (a *memaccessor) Stat() (fi os.FileInfo, err error) {
i := a.fi
i.FSize = int64(len(a.b))
fi = i
return
}
// Sync implements Accessor. Specifically it synchronizes the memory and file images.
func (a *memaccessor) Sync() (err error) {
var n int
if n, err = a.f.WriteAt(a.b, 0); n != len(a.b) {
return
}
return a.f.Truncate(int64(len(a.b)))
}
func (a *memaccessor) Truncate(size int64) (err error) {
defer func() {
if e := recover(); e != nil {
err = e.(error)
}
}()
if size > math.MaxInt32 {
panic(errors.New("truncate: illegal size"))
}
a.b = a.b[:int(size)]
return
}
func (a *memaccessor) WriteAt(b []byte, off int64) (n int, err error) {
if off < 0 || off > math.MaxInt32 {
return -1, errors.New("WriteAt: illegal offset")
}
rq, fp, size := len(b), int(off), len(a.b)
if need := rq + fp; need > size {
if need <= cap(a.b) {
a.b = a.b[:need]
} else {
nb := make([]byte, need, 2*need)
copy(nb, a.b)
a.b = nb
}
}
copy(a.b[int(off):], b)
return
}

74
vendor/github.com/cznic/fileutil/storage/probe.go generated vendored Normal file
View File

@@ -0,0 +1,74 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package storage
import "sync/atomic"
// Probe collects usage statistics of the embeded Accessor.
// Probe itself IS an Accessor.
type Probe struct {
Accessor
Chain *Probe
OpsRd int64
OpsWr int64
BytesRd int64
BytesWr int64
SectorsRd int64 // Assuming 512 byte sector size
SectorsWr int64
}
// NewProbe returns a newly created probe which embedes the src Accessor.
// The retuned *Probe satisfies Accessor. if chain != nil then Reset()
// is cascaded down the chained Probes.
func NewProbe(src Accessor, chain *Probe) *Probe {
return &Probe{Accessor: src, Chain: chain}
}
func reset(n *int64) {
atomic.AddInt64(n, -atomic.AddInt64(n, 0))
}
// Reset zeroes the collected statistics of p.
func (p *Probe) Reset() {
if p.Chain != nil {
p.Chain.Reset()
}
reset(&p.OpsRd)
reset(&p.OpsWr)
reset(&p.BytesRd)
reset(&p.BytesWr)
reset(&p.SectorsRd)
reset(&p.SectorsWr)
}
func (p *Probe) ReadAt(b []byte, off int64) (n int, err error) {
n, err = p.Accessor.ReadAt(b, off)
atomic.AddInt64(&p.OpsRd, 1)
atomic.AddInt64(&p.BytesRd, int64(n))
if n <= 0 {
return
}
sectorFirst := off >> 9
sectorLast := (off + int64(n) - 1) >> 9
atomic.AddInt64(&p.SectorsRd, sectorLast-sectorFirst+1)
return
}
func (p *Probe) WriteAt(b []byte, off int64) (n int, err error) {
n, err = p.Accessor.WriteAt(b, off)
atomic.AddInt64(&p.OpsWr, 1)
atomic.AddInt64(&p.BytesWr, int64(n))
if n <= 0 {
return
}
sectorFirst := off >> 9
sectorLast := (off + int64(n) - 1) >> 9
atomic.AddInt64(&p.SectorsWr, sectorLast-sectorFirst+1)
return
}

141
vendor/github.com/cznic/fileutil/storage/storage.go generated vendored Normal file
View File

@@ -0,0 +1,141 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
// WIP: Package storage defines and implements storage providers and store accessors.
package storage
import (
"os"
"sync"
"time"
)
// FileInfo is a type implementing os.FileInfo which has setable fields, like
// the older os.FileInfo used to have. It is used wehere e.g. the Size is
// needed to be faked (encapsulated/memory only file, file cache, etc.).
type FileInfo struct {
FName string // base name of the file
FSize int64 // length in bytes
FMode os.FileMode // file mode bits
FModTime time.Time // modification time
FIsDir bool // abbreviation for Mode().IsDir()
sys interface{} // underlying data source (can be nil)
}
// NewFileInfo creates FileInfo from os.FileInfo fi.
func NewFileInfo(fi os.FileInfo, sys interface{}) *FileInfo {
return &FileInfo{fi.Name(), fi.Size(), fi.Mode(), fi.ModTime(), fi.IsDir(), sys}
}
// Implementation of os.FileInfo
func (fi *FileInfo) Name() string {
return fi.FName
}
// Implementation of os.FileInfo
func (fi *FileInfo) Size() int64 {
return fi.FSize
}
// Implementation of os.FileInfo
func (fi *FileInfo) Mode() os.FileMode {
return fi.FMode
}
// Implementation of os.FileInfo
func (fi *FileInfo) ModTime() time.Time {
return fi.FModTime
}
// Implementation of os.FileInfo
func (fi *FileInfo) IsDir() bool {
return fi.FIsDir
}
func (fi *FileInfo) Sys() interface{} {
return fi.sys
}
// Accessor provides I/O methods to access a store.
type Accessor interface {
// Close closes the store, rendering it unusable for I/O. It returns an
// error, if any.
Close() error
// Name returns the name of the file as presented to Open.
Name() string
// ReadAt reads len(b) bytes from the store starting at byte offset off.
// It returns the number of bytes read and the error, if any.
// EOF is signaled by a zero count with err set to os.EOF.
// ReadAt always returns a non-nil Error when n != len(b).
ReadAt(b []byte, off int64) (n int, err error)
// Stat returns the FileInfo structure describing the store. It returns
// the os.FileInfo and an error, if any.
Stat() (fi os.FileInfo, err error)
// Sync commits the current contents of the store to stable storage.
// Typically, this means flushing the file system's in-memory copy of
// recently written data to disk.
Sync() (err error)
// Truncate changes the size of the store. It does not change the I/O
// offset.
Truncate(size int64) error
// WriteAt writes len(b) bytes to the store starting at byte offset off.
// It returns the number of bytes written and an error, if any.
// WriteAt returns a non-nil Error when n != len(b).
WriteAt(b []byte, off int64) (n int, err error)
// Before every [structural] change of a store the BeginUpdate is to be
// called and paired with EndUpdate after the change makes the store's
// state consistent again. Invocations of BeginUpdate may nest. On
// invoking the last non nested EndUpdate an implicit "commit" should
// be performed by the store/provider. The concrete mechanism is
// unspecified. It could be for example a write-ahead log. Stores may
// implement BeginUpdate and EndUpdate as a (documented) no op.
BeginUpdate() error
EndUpdate() error
}
// Mutate is a helper/wrapper for executing f in between a.BeginUpdate and
// a.EndUpdate. Any parameters and/or return values except an error should be
// captured by a function literal passed as f. The returned err is either nil
// or the first non nil error returned from the sequence of execution:
// BeginUpdate, [f,] EndUpdate. The pair BeginUpdate/EndUpdate *is* invoked
// always regardles of any possible errors produced. Mutate doesn't handle
// panic, it should be used only with a function [literal] which doesn't panic.
// Otherwise the pairing of BeginUpdate/EndUpdate is not guaranteed.
//
// NOTE: If BeginUpdate, which is invoked before f, returns a non-nil error,
// then f is not invoked at all (but EndUpdate still is).
func Mutate(a Accessor, f func() error) (err error) {
defer func() {
if e := a.EndUpdate(); e != nil && err == nil {
err = e
}
}()
if err = a.BeginUpdate(); err != nil {
return
}
return f()
}
// LockedMutate wraps Mutate in yet another layer consisting of a
// l.Lock/l.Unlock pair. All other limitations apply as in Mutate, e.g. no
// panics are allowed to happen - otherwise no guarantees can be made about
// Unlock matching the Lock.
func LockedMutate(a Accessor, l sync.Locker, f func() error) (err error) {
l.Lock()
defer l.Unlock()
return Mutate(a, f)
}

13
vendor/github.com/cznic/fileutil/storage/test_deps.go generated vendored Normal file
View File

@@ -0,0 +1,13 @@
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package storage
// Pull test dependencies too.
// Enables easy 'go test X' after 'go get X'
import (
// nothing yet
)

13
vendor/github.com/cznic/fileutil/test_deps.go generated vendored Normal file
View File

@@ -0,0 +1,13 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package fileutil
// Pull test dependencies too.
// Enables easy 'go test X' after 'go get X'
import (
// nothing yet
)