* go mod init; rm -rf vendor * tweak proto files and generation * go mod vendor * clean up build.go * protobuf literals in tests * downgrade gogo/protobuf
This commit is contained in:
Jakob Borg
12
vendor/github.com/chmduquesne/rollinghash/.travis.yml
generated
vendored
Normal file
12
vendor/github.com/chmduquesne/rollinghash/.travis.yml
generated
vendored
Normal file
@@ -0,0 +1,12 @@
|
||||
language: go
|
||||
go:
|
||||
- "1.10"
|
||||
- "1.9"
|
||||
- "1.8"
|
||||
- "1.7"
|
||||
before_install:
|
||||
- go get github.com/mattn/goveralls
|
||||
script:
|
||||
- go test -v ./...
|
||||
- go test -bench=. ./...
|
||||
- $HOME/gopath/bin/goveralls -package=./... -service=travis-ci
|
||||
120
vendor/github.com/chmduquesne/rollinghash/README.md
generated
vendored
Normal file
120
vendor/github.com/chmduquesne/rollinghash/README.md
generated
vendored
Normal file
@@ -0,0 +1,120 @@
|
||||
[](https://travis-ci.org/chmduquesne/rollinghash)
|
||||
[](https://coveralls.io/github/chmduquesne/rollinghash?branch=master)
|
||||
[](https://godoc.org/github.com/chmduquesne/rollinghash)
|
||||
|
||||
|
||||
Rolling Hashes
|
||||
==============
|
||||
|
||||
Philosophy
|
||||
----------
|
||||
|
||||
This package contains several various rolling hashes for you to play with
|
||||
crazy ideas. The API design philosophy is to stick as closely as possible
|
||||
to the interface provided by the builtin hash package (the hashes
|
||||
implemented here are effectively drop-in replacements for their builtin
|
||||
counterparts), while providing simultaneously the highest speed and
|
||||
simplicity.
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
A [`rollinghash.Hash`](https://godoc.org/github.com/chmduquesne/rollinghash#Hash)
|
||||
is just a [`hash.Hash`](https://golang.org/pkg/hash/#Hash) which
|
||||
implements the
|
||||
[`Roller`](https://godoc.org/github.com/chmduquesne/rollinghash#Roller)
|
||||
interface. Here is how it is typically used:
|
||||
|
||||
```golang
|
||||
data := []byte("here is some data to roll on")
|
||||
h := buzhash64.New()
|
||||
n := 16
|
||||
|
||||
// Initialize the rolling window
|
||||
h.Write(data[:n])
|
||||
|
||||
for _, c := range(data[n:]) {
|
||||
|
||||
// Slide the window and update the hash
|
||||
h.Roll(c)
|
||||
|
||||
// Get the updated hash value
|
||||
fmt.Println(h.Sum64())
|
||||
}
|
||||
```
|
||||
|
||||
Gotchas
|
||||
-------
|
||||
|
||||
The rolling window MUST be initialized by calling `Write` first (which
|
||||
saves a copy). The byte leaving the rolling window is inferred from the
|
||||
internal copy of the rolling window, which is updated with every call to
|
||||
`Roll`.
|
||||
|
||||
If you want your code to run at the highest speed, do NOT cast the result
|
||||
of a `New()` as a rollinghash.Hash. Instead, use the native type returned
|
||||
by `New()`. This is because the go compiler cannot inline calls from an
|
||||
interface. When later you call Roll(), the native type call will be
|
||||
inlined by the compiler, but not the casted type call.
|
||||
|
||||
```golang
|
||||
var h1 rollinghash.Hash
|
||||
h1 = buzhash32.New()
|
||||
h2 := buzhash32.New()
|
||||
|
||||
[...]
|
||||
|
||||
h1.Roll(b) // Not inlined (slow)
|
||||
h2.Roll(b) // inlined (fast)
|
||||
```
|
||||
|
||||
What's new in v4
|
||||
----------------
|
||||
|
||||
In v4:
|
||||
|
||||
* `Write` has become fully consistent with `hash.Hash`. As opposed to
|
||||
previous versions, where writing data would reinitialize the window, it
|
||||
now appends this data to the existing window. In order to reset the
|
||||
window, one should instead use the `Reset` method.
|
||||
|
||||
* Calling `Roll` on an empty window is considered a bug, and now triggers
|
||||
a panic.
|
||||
|
||||
Brief reminder of the behaviors in previous versions:
|
||||
|
||||
* From v0.x.x to v2.x.x: `Roll` returns an error for an empty window.
|
||||
`Write` reinitializes the rolling window.
|
||||
|
||||
* v3.x.x : `Roll` does not return anything. `Write` still reinitializes
|
||||
the rolling window. The rolling window always has a minimum size of 1,
|
||||
which yields wrong results when using roll before having initialized the
|
||||
window.
|
||||
|
||||
Go versions
|
||||
-----------
|
||||
|
||||
The `RabinKarp64` rollinghash does not yield consistent results before
|
||||
go1.7. This is because it uses `Rand.Read()` from the builtin `math/rand`.
|
||||
This function was [fixed in go
|
||||
1.7](https://golang.org/doc/go1.7#math_rand) to produce a consistent
|
||||
stream of bytes that is independant of the size of the input buffer. If
|
||||
you depend on this hash, it is strongly recommended to stick to versions
|
||||
of go superior to 1.7.
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
This code is delivered to you under the terms of the MIT public license,
|
||||
except the `rabinkarp64` subpackage, which has been adapted from
|
||||
[restic](https://github.com/restic/chunker) (BSD 2-clause "Simplified").
|
||||
|
||||
Notable users
|
||||
-------------
|
||||
|
||||
* [syncthing](https://syncthing.net/), a decentralized synchronisation
|
||||
solution
|
||||
* [muscato](https://github.com/kshedden/muscato), a genome analysis tool
|
||||
|
||||
If you are using this in production or for research, let me know and I
|
||||
will happily put a link here!
|
||||
140
vendor/github.com/chmduquesne/rollinghash/roll/main.go
generated
vendored
140
vendor/github.com/chmduquesne/rollinghash/roll/main.go
generated
vendored
@@ -1,140 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"hash"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"runtime/pprof"
|
||||
"time"
|
||||
|
||||
"code.cloudfoundry.org/bytefmt"
|
||||
//rollsum "github.com/chmduquesne/rollinghash/adler32"
|
||||
//rollsum "github.com/chmduquesne/rollinghash/buzhash32"
|
||||
rollsum "github.com/chmduquesne/rollinghash/buzhash64"
|
||||
//rollsum "github.com/chmduquesne/rollinghash/bozo32"
|
||||
)
|
||||
|
||||
const (
|
||||
KiB = 1024
|
||||
MiB = 1024 * KiB
|
||||
GiB = 1024 * MiB
|
||||
|
||||
clearscreen = "\033[2J\033[1;1H"
|
||||
clearline = "\x1b[2K"
|
||||
)
|
||||
|
||||
func genMasks() (res []uint64) {
|
||||
res = make([]uint64, 64)
|
||||
ones := ^uint64(0) // 0xffffffffffffffff
|
||||
for i := 0; i < 64; i++ {
|
||||
res[i] = ones >> uint(63-i)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Gets the hash sum as a uint64
|
||||
func sum64(h hash.Hash) (res uint64) {
|
||||
buf := make([]byte, 0, 8)
|
||||
s := h.Sum(buf)
|
||||
for _, b := range s {
|
||||
res <<= 8
|
||||
res |= uint64(b)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func main() {
|
||||
cpuprofile := flag.String("cpuprofile", "", "write cpu profile to file")
|
||||
dostats := flag.Bool("stats", false, "Do some stats about the rolling sum")
|
||||
size := flag.String("size", "256M", "How much data to read")
|
||||
flag.Parse()
|
||||
|
||||
if *cpuprofile != "" {
|
||||
f, err := os.Create(*cpuprofile)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
pprof.StartCPUProfile(f)
|
||||
defer pprof.StopCPUProfile()
|
||||
}
|
||||
|
||||
fileSize, err := bytefmt.ToBytes(*size)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
bufsize := 16 * MiB
|
||||
buf := make([]byte, bufsize)
|
||||
t := time.Now()
|
||||
|
||||
f, err := os.Open("/dev/urandom")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer func() {
|
||||
if err := f.Close(); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}()
|
||||
|
||||
io.ReadFull(f, buf)
|
||||
|
||||
roll := rollsum.New()
|
||||
roll.Write(buf[:64])
|
||||
|
||||
masks := genMasks()
|
||||
hits := make(map[uint64]uint64)
|
||||
for _, m := range masks {
|
||||
hits[m] = 0
|
||||
}
|
||||
|
||||
n := uint64(0)
|
||||
k := 0
|
||||
for n < fileSize {
|
||||
if k >= bufsize {
|
||||
status := fmt.Sprintf("Byte count: %s", bytefmt.ByteSize(n))
|
||||
if *dostats {
|
||||
fmt.Printf(clearscreen)
|
||||
fmt.Println(status)
|
||||
for i, m := range masks {
|
||||
frequency := "NaN"
|
||||
if hits[m] != 0 {
|
||||
frequency = bytefmt.ByteSize(n / hits[m])
|
||||
}
|
||||
fmt.Printf("0x%016x (%02d bits): every %s\n", m, i+1, frequency)
|
||||
}
|
||||
} else {
|
||||
fmt.Printf(clearline)
|
||||
fmt.Printf(status)
|
||||
fmt.Printf("\r")
|
||||
}
|
||||
_, err := io.ReadFull(f, buf)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
k = 0
|
||||
}
|
||||
roll.Roll(buf[k])
|
||||
if *dostats {
|
||||
s := sum64(roll)
|
||||
for _, m := range masks {
|
||||
if s&m == m {
|
||||
hits[m] += 1
|
||||
} else {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
k++
|
||||
n++
|
||||
}
|
||||
duration := time.Since(t)
|
||||
fmt.Printf("Rolled %s of data in %v (%s/s).\n",
|
||||
bytefmt.ByteSize(n),
|
||||
duration,
|
||||
bytefmt.ByteSize(n*1e9/uint64(duration)),
|
||||
)
|
||||
}
|
||||
Reference in New Issue
Block a user