Crayfish Common CRC

Release 2.1.0Java 8+BSD 3-Clause

Fast, zero-dependency CRC library for Java

Every CRC from CRC-3 to CRC-64 of the RevEng catalogue — CRC-32, CRC-32C, CRC-16/MODBUS, CRC-64/XZ and 108 more — and any custom one up to 64 bits.

Get started GitHub Javadoc

QuickStart.java
byte[] data = "123456789".getBytes(StandardCharsets.US_ASCII);

long modbus = CrcModel.of("CRC-16/MODBUS").value(data);   // 0x4B37
long crc32c = CrcModel.of("CRC-32C").value(data);         // 0xE3069283
long xz     = CrcModel.of("CRC-64/XZ").value(data);       // 0x995DC9BBDF1939FA
112
algorithms of the catalogue
186
names with aliases
3…64
bits, 21 widths predefined
0
runtime dependencies
8…25
JDK versions built and tested

Install

The library is on Maven Central: one jar under 70 KB, no dependencies at run time.

<dependency>
    <groupId>com.github.sftwnd.crayfish.common</groupId>
    <artifactId>crayfish-common-crc</artifactId>
    <version>2.1.0</version>
</dependency>
implementation 'com.github.sftwnd.crayfish.common:crayfish-common-crc:2.1.0'
implementation("com.github.sftwnd.crayfish.common:crayfish-common-crc:2.1.0")

On Java 9+ the jar is the named module com.github.sftwnd.crayfish_common_crc; OSGi bundle headers are included. Coming from 1.x? See Migrating from 1.x.

What it does

More than a table of polynomials: a CRC is a value you can keep, combine, resume and send.

  • Fast by default

    Table-driven slicing-by-8 for every model, not only for CRC-32: 8 bytes per step, 2–4 times faster than one byte per step on buffers from 256 bytes. A CRC up to 32 bits wide is calculated in an int register.

  • CPU instructions on request

    CRC-32, CRC-32/JAMCRC and CRC-32C can be calculated by java.util.zip.CRC32 and CRC32C, which the JVM runs on the CRC instructions of the CPU — with the whole API of the library.

  • Immutable CRCs

    A CRC is a value: an update or a combination gives a new CRC, so a CRC can be shared between threads and kept as the state of a calculation.

  • Combine and resume

    Join the CRCs of parts calculated in parallel threads or on other nodes, like zlib crc32_combine. Continue from a saved value and length, also of 2 GiB and more.

  • Streams

    A java.util.zip.Checksum of any model for CheckedInputStream, CheckedOutputStream and other code written for CRC32; the CRC of an InputStream or a channel read to its end.

  • Codewords

    Build and check data followed by its CRC, in the byte order of the model or of a format such as PNG. The bytes of a CRC in the byte order of a protocol, and its hexadecimal digits.

  • CRCs of bits

    The CRC of data that is not a whole number of bytes, such as the frames of CAN, FlexRay and USB.

  • Find the algorithm

    Give samples of data with their CRCs and get the algorithms of the catalogue that produce them — to find out the CRC of a device or of a file format.

  • Names and aliases

    A model is chosen by the name or by any alias of the catalogue, in any case of its letters: CRC-32, PKZIP and CRC-32/XZ are one model.

  • Any CRC up to 64 bits

    Width 1…64, any poly, init, refin, refout and xorout: by the parameters, with a builder or from a parameter line of the catalogue.

  • Memory on demand

    Lookup tables are built on the first calculation and collected when no model uses them any more. No thread of its own, so the library is unloaded with a redeployed application.

  • Java 8+ and modular

    Runs on Java 8; on Java 9+ it is a named JPMS module of a multi-release jar, with OSGi bundle headers. Two builds of the same sources give the same jar byte for byte.

Verified. Every predefined model is tested against its RevEng check value and residue and against a bit-by-bit reference implementation.

In code

The whole API is a model, a CRC and a checksum. The examples are those of the README, which explains every one of them.

The value of a CRC, an immutable CRC to continue or to combine, its hexadecimal digits and its bytes in the byte order of a protocol. More

CrcModel model = CrcModel.of("CRC-16/ARC");
long value = model.value(data);                                  // the CRC value only, no CRC object
long part = model.value(data, offset, length);                   // a part of the data
CRC crc = model.crc().update(part1).update(part2, length2);      // part1 followed by the first length2 bytes of part2

String crc32 = CrcModel.of("CRC-32").crc(data).hex();            // cbf43926 for "123456789"
byte[] modbus = CrcModel.of("CRC-16/MODBUS").crc(data).bytes(ByteOrder.LITTLE_ENDIAN);   // 37 4b (0x4b37)

The CRC of parts calculated independently. The cost does not depend on the size of the data. More

CRC first = model.crc(part1);        // calculated independently,
CRC second = model.crc(part2);       // for example in parallel threads
CRC both = first.combine(second);    // the CRC of part1 followed by part2

// only the values are kept, like zlib crc32_combine(crc1, crc2, len2):
long value = model.combine(first.value(), second.value(), second.length());

Continue a calculation from a saved CRC value and length, as a CRC or as a java.util.zip.Checksum. More

long savedValue = crc.value();
long savedLength = crc.length();
// later, also in another process
CRC resumed = model.crc(savedValue, savedLength).update(moreData);

// append to a file whose CRC is kept
CrcChecksum checksum = model.crc(savedValue, savedLength).checksum();
try (OutputStream file = Files.newOutputStream(path, StandardOpenOption.APPEND);
     CheckedOutputStream out = new CheckedOutputStream(file, checksum)) {
    out.write(moreData);               // after the bytes written before
}
CRC whole = checksum.crc();            // the CRC of the whole file

Any CRC works with the JDK classes made for CRC32; a stream or a channel needs no loop. More

CrcModel model = CrcModel.of("CRC-32C");
try (CheckedOutputStream out = new CheckedOutputStream(Files.newOutputStream(path), model.checksum())) {
    out.write(header);                 // the parts in the order they are written
    out.write(body, 0, bodyLength);
    long value = out.getChecksum().getValue();   // the CRC value of all bytes written
}

try (InputStream in = Files.newInputStream(path)) {
    CRC crc = model.crc(in);                        // all the bytes of the stream
}
try (FileChannel channel = FileChannel.open(path)) {
    CRC crc = model.crc(header).update(channel);    // the header followed by all the bytes of the channel
}

A codeword is data followed by its CRC, as a frame of a protocol or a block of a file carries them. More

CrcModel modbus = CrcModel.of("CRC-16/MODBUS");
byte[] frame = modbus.codeword(pdu);               // the PDU followed by its CRC, low byte first
boolean valid = modbus.isValid(frame);             // true
boolean inside = modbus.isValid(buffer, 4, 10);    // a codeword of 10 bytes at the offset 4 of a buffer

// PNG stores the CRC-32 of a chunk most significant byte first
CrcModel crc32 = CrcModel.of("CRC-32");
byte[] type = "IEND".getBytes(StandardCharsets.US_ASCII);
byte[] iend = crc32.codeword(type, ByteOrder.BIG_ENDIAN);               // 49 45 4e 44 ae 42 60 82
boolean chunk = crc32.isValid(data, 4, n + 8, ByteOrder.BIG_ENDIAN);    // the type, n bytes of data and the CRC

Some protocols calculate the CRC of bits that are not a whole number of bytes. More

// CAN: the 19 bits before the data and the data, packed most significant bit first
long can = CrcModel.of("CRC-15/CAN").valueOfBits(frameBits, 19 + 8 * dlc);
// FlexRay: the 20 bits of the header in a long
long header = CrcModel.of("CRC-11/FLEXRAY").valueOfBits(headerBits, 20);
// USB: the 11 bits of a token, the first bit in bit 0 (refin)
long token = CrcModel.of("CRC-5/USB").valueOfBits(address | endpoint << 7, 11);

A CRC outside the catalogue: by its parameters, with a builder or from a parameter line. More

CrcModel other = CrcModel.of(16, 0x1234, 0x0, true, true, 0x0);    // CRC-16/P1234_I0_RI_RO

CrcModel crc32 = CrcModel.builder(32)   // the width
        .poly(0x04C11DB7L)              // required
        .reflected(true)                // required: refin and refout, or refin(...) and refout(...)
        .init(0xFFFFFFFFL)              // 0 if not set
        .xorout(0xFFFFFFFFL)            // 0 if not set
        .check(0xCBF43926L)             // optional: build() verifies it
        .build();                       // CRC-32/ISO-HDLC

CrcParameters arc = CrcParameters.parse("width=16 poly=0x8005 init=0x0000 refin=true refout=true xorout=0x0000");

Which CRC does a device or a file format use? Take samples of its data with their CRCs. More

Set<StandardCrc> found = StandardCrc.matching(data, 0x4B37L);                    // the algorithms that give the value
found.retainAll(StandardCrc.matching(frame, 0, frame.length - 2, crcOfFrame));   // another sample narrows them down
if (found.size() == 1) {                                                         // the algorithm of the samples
    CrcModel model = found.iterator().next().model();
}

CRC-32 and CRC-32C on the CRC instructions of the CPU, chosen per model or by a default: -Dcom.github.sftwnd.crayfish.common.crc.jdk=true. More

CrcModel jdk = CrcModel.of("CRC-32", true);                  // java.util.zip.CRC32
CrcModel tables = CrcModel.of("CRC-32", false);              // the lookup tables of the library
CrcModel same = tables.withJdk(true);                        // the model jdk

CrcDescription.setJdk(true);                                 // the default of the models chosen without a choice
CrcModel byDefault = CrcModel.of("CRC-32C");                 // java.util.zip.CRC32C on Java 9+

The catalogue, by name

112 algorithms of 21 widths. Every name and alias is accepted by CrcModel.of(). Pick one to see its parameters and to calculate a CRC right here.

The bytes of the text in UTF-8

CRC-16/MODBUS of 9 bytes

0x4B37

long crc = CrcModel.of("CRC-16/MODBUS")
        .value(data);

Calculated in your browser, bit by bit, from the parameters of the catalogue. The library gives the same values with its lookup tables.

112 algorithms

CRC-3 2

CRC-4 2

CRC-5 3

CRC-6 5

CRC-7 3

CRC-8 20

CRC-10 3

CRC-11 2

CRC-12 4

CRC-13 1

CRC-14 2

CRC-15 2

CRC-16 31

CRC-17 1

CRC-21 1

CRC-24 8

CRC-30 1

CRC-31 1

CRC-32 12

CRC-40 1

CRC-64 7

CRC-82/DARC of the catalogue is not predefined: a CRC value is a long, the width is limited to 64 bits.

Speed

Measured with JMH. The full comparison with 1.2.0 and 2.0.0 on JDK 8, 21 and 25 is on the wiki page Speed of 2.1.0.

  • 2–4×slicing-by-8 against one byte per step, on buffers from 256 bytes
  • 1.1–1.7×2.1.0 against 2.0 for CRCs up to 32 bits wide: the register of an int
  • 16×CRC-32C of 4 KB on CPU instructions through the JDK: 0.13 µs instead of 2.14 µs
The average time of model.value(data): the lookup tables of the library → the class of the JDK (Intel Core i9-9980HK)
Model, JDK16 B256 B4 KB1 MB
CRC-32, JDK 2513.8 → 16.0 ns141 → 62 ns2.24 → 0.20 µs515 → 42 µs
CRC-32C, JDK 2513.8 → 5.5 ns145 → 13 ns2.14 → 0.13 µs513 → 41 µs
CRC-32, JDK 818.8 → 19.6 ns141 → 59 ns2.41 → 0.20 µs615 → 39 µs

The lookup tables are the default; the classes of the JDK are chosen per model or by a system property. The details, and what the speed depends on, are in the README: Speed and CRC-32 and CRC-32C of the JDK.

Add it to your build

com.github.sftwnd.crayfish.common:crayfish-common-crc:2.1.0

Read the documentation README Report an issue