2017-09-17 20:30:41 +08:00
|
|
|
libmdbx
|
2017-01-09 02:48:00 +08:00
|
|
|
======================================
|
2018-01-18 03:19:59 +08:00
|
|
|
**Revised and extended descendant of [Symas LMDB](https://symas.com/lmdb/).**
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-01-18 03:19:59 +08:00
|
|
|
*The Future will be positive.*
|
2017-07-30 05:58:11 +08:00
|
|
|
[![Build Status](https://travis-ci.org/leo-yuriev/libmdbx.svg?branch=master)](https://travis-ci.org/leo-yuriev/libmdbx)
|
|
|
|
[![Build status](https://ci.appveyor.com/api/projects/status/ue94mlopn50dqiqg/branch/master?svg=true)](https://ci.appveyor.com/project/leo-yuriev/libmdbx/branch/master)
|
2017-06-12 20:41:36 +08:00
|
|
|
[![Coverity Scan Status](https://scan.coverity.com/projects/12915/badge.svg)](https://scan.coverity.com/projects/reopen-libmdbx)
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-05-10 19:05:48 +08:00
|
|
|
## Project Status for now
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- The stable versions
|
|
|
|
([_stable/0.0_](https://github.com/leo-yuriev/libmdbx/tree/stable/0.0)
|
|
|
|
and
|
|
|
|
[_stable/0.1_](https://github.com/leo-yuriev/libmdbx/tree/stable/0.1)
|
|
|
|
branches) of _MDBX_ are frozen, i.e. no new features or API changes, but
|
|
|
|
only bug fixes.
|
|
|
|
|
|
|
|
- The next version
|
|
|
|
([_devel_](https://github.com/leo-yuriev/libmdbx/tree/devel) branch)
|
|
|
|
**is under active non-public development**, i.e. current API and set of
|
|
|
|
features are extreme volatile.
|
|
|
|
|
|
|
|
- The immediate goal of development is formation of the stable API and
|
|
|
|
the stable internal database format, which allows realise all PLANNED
|
|
|
|
FEATURES:
|
2018-05-25 05:00:58 +08:00
|
|
|
1. Integrity check by [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree);
|
|
|
|
2. Support for [raw block devices](https://en.wikipedia.org/wiki/Raw_device);
|
2018-05-22 17:01:35 +08:00
|
|
|
3. Separate place (HDD) for large data items;
|
2018-05-25 05:00:58 +08:00
|
|
|
4. Using "[Roaring bitmaps](http://roaringbitmap.org/about/)" inside garbage collector;
|
2018-05-29 07:04:51 +08:00
|
|
|
5. Non-sequential reclaiming, like PostgreSQL's [Vacuum](https://www.postgresql.org/docs/9.1/static/sql-vacuum.html);
|
2018-05-25 05:00:58 +08:00
|
|
|
6. [Asynchronous lazy data flushing](https://sites.fas.harvard.edu/~cs265/papers/kathuria-2008.pdf) to disk(s);
|
2018-05-22 17:01:35 +08:00
|
|
|
7. etc...
|
2018-05-10 19:05:48 +08:00
|
|
|
|
2018-10-17 09:42:26 +08:00
|
|
|
Don't miss libmdbx for other runtimes.
|
|
|
|
|
|
|
|
| Runtime | GitHub | Author |
|
|
|
|
| ------------- | ------------- | ------------- |
|
|
|
|
| JVM | [mdbxjni](https://github.com/castortech/mdbxjni) | [Castor Technologies](https://castortech.com/) |
|
|
|
|
| .NET | [mdbx.NET](https://github.com/wangjia184/mdbx.NET) | [Jerry Wang](https://github.com/wangjia184) |
|
2018-06-19 04:04:49 +08:00
|
|
|
|
2018-05-10 19:05:48 +08:00
|
|
|
-----
|
2018-03-26 19:17:35 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
Nowadays MDBX intended for Linux, and support Windows (since Windows
|
|
|
|
Server 2008) as a complementary platform. Support for other OS could be
|
|
|
|
implemented on commercial basis. However such enhancements (i.e. pull
|
|
|
|
requests) could be accepted in mainstream only when corresponding public
|
|
|
|
and free Continuous Integration service will be available.
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2018-01-18 03:19:59 +08:00
|
|
|
## Contents
|
|
|
|
- [Overview](#overview)
|
|
|
|
- [Comparison with other DBs](#comparison-with-other-dbs)
|
2018-05-29 07:04:51 +08:00
|
|
|
- [History & Acknowledgments](#history)
|
2018-01-18 03:19:59 +08:00
|
|
|
- [Main features](#main-features)
|
2018-07-30 07:55:13 +08:00
|
|
|
- [Improvements over LMDB](#improvements-over-lmdb)
|
|
|
|
- [Gotchas](#gotchas)
|
|
|
|
- [Long-time read transactions problem](#long-time-read-transactions-problem)
|
|
|
|
- [Data safety in async-write-mode](#data-safety-in-async-write-mode)
|
2018-01-18 20:47:17 +08:00
|
|
|
- [Performance comparison](#performance-comparison)
|
|
|
|
- [Integral performance](#integral-performance)
|
2018-01-18 03:19:59 +08:00
|
|
|
- [Read scalability](#read-scalability)
|
|
|
|
- [Sync-write mode](#sync-write-mode)
|
|
|
|
- [Lazy-write mode](#lazy-write-mode)
|
|
|
|
- [Async-write mode](#async-write-mode)
|
|
|
|
- [Cost comparison](#cost-comparison)
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2017-07-21 06:59:05 +08:00
|
|
|
|
2018-01-18 03:19:59 +08:00
|
|
|
## Overview
|
2018-07-30 07:55:13 +08:00
|
|
|
_libmdbx_ is an embedded lightweight key-value database engine oriented
|
|
|
|
for performance under Linux and Windows.
|
2017-07-21 06:59:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
_libmdbx_ allows multiple processes to read and update several key-value
|
|
|
|
tables concurrently, while being
|
|
|
|
[ACID](https://en.wikipedia.org/wiki/ACID)-compliant, with minimal
|
|
|
|
overhead and operation cost of Olog(N).
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2018-01-18 03:19:59 +08:00
|
|
|
_libmdbx_ provides
|
2018-07-30 07:55:13 +08:00
|
|
|
[serializability](https://en.wikipedia.org/wiki/Serializability) and
|
|
|
|
consistency of data after crash. Read-write transactions don't block
|
|
|
|
read-only transactions and are
|
|
|
|
[serialized](https://en.wikipedia.org/wiki/Serializability) by
|
|
|
|
[mutex](https://en.wikipedia.org/wiki/Mutual_exclusion).
|
|
|
|
|
|
|
|
_libmdbx_
|
|
|
|
[wait-free](https://en.wikipedia.org/wiki/Non-blocking_algorithm#Wait-freedom)
|
|
|
|
provides parallel read transactions without atomic operations or
|
|
|
|
synchronization primitives.
|
|
|
|
|
|
|
|
_libmdbx_ uses [B+Trees](https://en.wikipedia.org/wiki/B%2B_tree) and
|
|
|
|
[mmap](https://en.wikipedia.org/wiki/Memory-mapped_file), doesn't use
|
|
|
|
[WAL](https://en.wikipedia.org/wiki/Write-ahead_logging). This might
|
|
|
|
have caveats for some workloads.
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2018-01-18 03:19:59 +08:00
|
|
|
### Comparison with other DBs
|
2018-07-30 07:55:13 +08:00
|
|
|
Because _libmdbx_ is currently overhauled, I think it's better to just
|
|
|
|
link [chapter of Comparison with other
|
|
|
|
databases](https://github.com/coreos/bbolt#comparison-with-other-databases)
|
|
|
|
here.
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2018-01-18 03:19:59 +08:00
|
|
|
### History
|
2018-07-30 07:55:13 +08:00
|
|
|
The _libmdbx_ design is based on [Lightning Memory-Mapped
|
|
|
|
Database](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database).
|
|
|
|
Initial development was going in
|
|
|
|
[ReOpenLDAP](https://github.com/leo-yuriev/ReOpenLDAP) project, about a
|
|
|
|
year later it received separate development effort and in autumn 2015
|
|
|
|
was isolated to separate project, which was [presented at Highload++
|
|
|
|
2015 conference](http://www.highload.ru/2015/abstracts/1831.html).
|
|
|
|
|
|
|
|
Since early 2017 _libmdbx_ is used in [Fast PositiveTables](https://github.com/leo-yuriev/libfpta),
|
2018-01-18 20:47:17 +08:00
|
|
|
by [Positive Technologies](https://www.ptsecurity.com).
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-05-29 07:04:51 +08:00
|
|
|
#### Acknowledgments
|
2018-07-30 07:55:13 +08:00
|
|
|
Howard Chu (Symas Corporation) - the author of LMDB, from which
|
|
|
|
originated the MDBX in 2015.
|
2017-09-17 20:07:32 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
Martin Hedenfalk <martin@bzero.se> - the author of `btree.c` code, which
|
|
|
|
was used for begin development of LMDB.
|
2017-09-17 20:07:32 +08:00
|
|
|
|
|
|
|
|
2018-01-18 03:19:59 +08:00
|
|
|
Main features
|
2018-01-18 20:47:17 +08:00
|
|
|
=============
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-01-18 20:47:17 +08:00
|
|
|
_libmdbx_ inherits all keys features and characteristics from
|
2018-01-18 03:19:59 +08:00
|
|
|
[LMDB](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database):
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
1. Data is stored in ordered map, keys are always sorted, range lookups
|
|
|
|
are supported.
|
|
|
|
|
|
|
|
2. Data is [mmaped](https://en.wikipedia.org/wiki/Memory-mapped_file) to
|
|
|
|
memory of each worker DB process, read transactions are zero-copy.
|
|
|
|
|
|
|
|
3. Transactions are
|
|
|
|
[ACID](https://en.wikipedia.org/wiki/ACID)-compliant, thanks to
|
|
|
|
[MVCC](https://en.wikipedia.org/wiki/Multiversion_concurrency_control)
|
|
|
|
and [CoW](https://en.wikipedia.org/wiki/Copy-on-write). Writes are
|
|
|
|
strongly serialized and aren't blocked by reads, transactions can't
|
|
|
|
conflict with each other. Reads are guaranteed to get only commited data
|
|
|
|
([relaxing serializability](https://en.wikipedia.org/wiki/Serializability#Relaxing_serializability)).
|
|
|
|
|
|
|
|
4. Reads and queries are
|
|
|
|
[non-blocking](https://en.wikipedia.org/wiki/Non-blocking_algorithm),
|
|
|
|
don't use [atomic
|
|
|
|
operations](https://en.wikipedia.org/wiki/Linearizability#High-level_atomic_operations).
|
|
|
|
Readers don't block each other and aren't blocked by writers. Read
|
|
|
|
performance scales linearly with CPU core count.
|
|
|
|
> Though "connect to DB" (start of first read transaction in thread) and
|
|
|
|
> "disconnect from DB" (shutdown or thread termination) requires to
|
|
|
|
> acquire a lock to register/unregister current thread from "readers
|
|
|
|
> table"
|
|
|
|
|
|
|
|
5. Keys with multiple values are stored efficiently without key
|
|
|
|
duplication, sorted by value, including integers (reasonable for
|
|
|
|
secondary indexes).
|
|
|
|
|
|
|
|
6. Efficient operation on short fixed length keys, including integer
|
|
|
|
ones.
|
|
|
|
|
|
|
|
7. [WAF](https://en.wikipedia.org/wiki/Write_amplification) (Write
|
|
|
|
Amplification Factor) и RAF (Read Amplification Factor) are Olog(N).
|
|
|
|
|
|
|
|
8. No [WAL](https://en.wikipedia.org/wiki/Write-ahead_logging) and
|
|
|
|
transaction journal. In case of a crash no recovery needed. No need for
|
|
|
|
regular maintenance. Backups can be made on the fly on working DB
|
|
|
|
without freezing writers.
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-01-18 03:19:59 +08:00
|
|
|
9. No custom memory management, all done with standard OS syscalls.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
--------------------------------------------------------------------------------
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
Improvements over LMDB
|
|
|
|
======================
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
1. `mdbx_chk` tool for DB integrity check.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
2. Automatic dynamic DB size management according to the parameters
|
|
|
|
specified by `mdbx_env_set_geometry()` function. Including including
|
|
|
|
growth step and truncation threshold, as well as the choice of page
|
|
|
|
size.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
3. Automatic returning of freed pages into unallocated space at the end
|
|
|
|
of database file with optionally automatic shrinking it. This reduces
|
|
|
|
amount of pages resides in RAM and circulated in disk I/O. In fact
|
|
|
|
_libmdbx_ constantly performs DB compactification, without spending
|
|
|
|
additional resources for that.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
4. Support for keys and values of zero length, including sorted
|
|
|
|
duplicates.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
5. Ability to assign up to 3 markers to commiting transaction with
|
|
|
|
`mdbx_canary_put()` and then get them in read transaction by
|
|
|
|
`mdbx_canary_get()`.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
6. Ability to update or delete record and get previous value via
|
|
|
|
`mdbx_replace()` Also can update specific multi-value.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
7. `LIFO RECLAIM` mode:
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
The newest pages are picked for reuse instead of the oldest. This allows
|
|
|
|
to minimize reclaim loop and make it execution time independent of total
|
|
|
|
page count.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
This results in OS kernel cache mechanisms working with maximum
|
|
|
|
efficiency. In case of using disk controllers or storages with
|
|
|
|
[BBWC](https://en.wikipedia.org/wiki/Disk_buffer#Write_acceleration)
|
|
|
|
this may greatly improve write performance.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
8. Sequence generation via `mdbx_dbi_sequence()`.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
9. `OOM-KICK` callback.
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
`mdbx_env_set_oomfunc()` allows to set a callback, which will be called
|
|
|
|
in the event of DB space exhausting during long-time read transaction in
|
|
|
|
parallel with extensive updating. Callback will be invoked with PID and
|
|
|
|
pthread_id of offending thread as parameters. Callback can do any of
|
|
|
|
these things to remedy the problem:
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* wait for read transaction to finish normally;
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* kill the offending process (signal 9), if separate process is doing
|
|
|
|
long-time read;
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* abort or restart offending read transaction if it's running in sibling
|
|
|
|
thread;
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* abort current write transaction with returning error code.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
10. Ability to open DB in exclusive mode with `MDBX_EXCLUSIVE` flag.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
11. Ability to get how far current read-only snapshot is from latest
|
|
|
|
version of the DB by `mdbx_txn_straggler()`.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
12. Ability to explicitly request update of present record without
|
|
|
|
creating new record. Implemented as `MDBX_CURRENT` flag for
|
|
|
|
`mdbx_put()`.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
13. Fixed `mdbx_cursor_count()`, which returns correct count of
|
|
|
|
duplicated for all table types and any cursor position.
|
2017-01-09 06:19:24 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
14. `mdbx_env_info()` to getting additional info, including number of
|
|
|
|
the oldest snapshot of DB, which is used by one of the readers.
|
2017-01-09 06:19:24 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
15. `mdbx_del()` doesn't ignore additional argument (specifier) `data`
|
|
|
|
for tables without duplicates (without flag `MDBX_DUPSORT`), if `data`
|
|
|
|
is not null then always uses it to verify record, which is being
|
|
|
|
deleted.
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
16. Ability to open dbi-table with simultaneous setup of comparators for
|
|
|
|
keys and values, via `mdbx_dbi_open_ex()`.
|
2017-01-09 06:19:24 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
17. `mdbx_is_dirty()`to find out if key or value is on dirty page, that
|
|
|
|
useful to avoid copy-out before updates.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
18. Correct update of current record in `MDBX_CURRENT` mode of
|
|
|
|
`mdbx_cursor_put()`, including sorted duplicated.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
19. Check if there is a row with data after current cursor position via
|
|
|
|
`mdbx_cursor_eof()`.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
20. Additional error code `MDBX_EMULTIVAL`, which is returned by
|
|
|
|
`mdbx_put()` and `mdbx_replace()` in case is ambiguous update or delete.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
21. Ability to get value by key and duplicates count by `mdbx_get_ex()`.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
22. Functions `mdbx_cursor_on_first()` and `mdbx_cursor_on_last()`,
|
|
|
|
which allows to know if cursor is currently on first or last position
|
|
|
|
respectively.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
23. Automatic creation of synchronization points (flush changes to
|
|
|
|
persistent storage) when changes reach set threshold (threshold can be
|
|
|
|
set by `mdbx_env_set_syncbytes()`).
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
24. Control over debugging and receiving of debugging messages via
|
|
|
|
`mdbx_setup_debug()`.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
25. Function `mdbx_env_pgwalk()` for page-walking all pages in DB.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
26. Three meta-pages instead of two, this allows to guarantee
|
|
|
|
consistently update weak sync-points without risking to corrupt last
|
|
|
|
steady sync-point.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
27. Guarantee of DB integrity in `WRITEMAP+MAPSYNC` mode:
|
|
|
|
> Current _libmdbx_ gives a choice of safe async-write mode (default)
|
|
|
|
> and `UTTERLY_NOSYNC` mode which may result in full
|
|
|
|
> DB corruption during system crash as with LMDB. For details see
|
|
|
|
> [Data safety in async-write mode](#data-safety-in-async-write-mode).
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
28. Ability to close DB in "dirty" state (without data flush and
|
|
|
|
creation of steady synchronization point) via `mdbx_env_close_ex()`.
|
2017-07-21 06:59:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
29. If read transaction is aborted via `mdbx_txn_abort()` or
|
|
|
|
`mdbx_txn_reset()` then DBI-handles, which were opened in it, aren't
|
|
|
|
closed or deleted. This allows to avoid several types of hard-to-debug
|
|
|
|
errors.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
30. All cursors in all read and write transactions can be reused by
|
|
|
|
`mdbx_cursor_renew()` and MUST be freed explicitly.
|
|
|
|
> ## Caution, please pay attention!
|
|
|
|
>
|
|
|
|
> This is the only change of API, which changes semantics of cursor management
|
|
|
|
> and can lead to memory leaks on misuse. This is a needed change as it eliminates ambiguity
|
|
|
|
> which helps to avoid such errors as:
|
|
|
|
> - use-after-free;
|
|
|
|
> - double-free;
|
|
|
|
> - memory corruption and segfaults.
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2017-07-04 17:07:05 +08:00
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
## Gotchas
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
1. At one moment there can be only one writer. But this allows to
|
|
|
|
serialize writes and eliminate any possibility of conflict or logical
|
|
|
|
errors during transaction rollback.
|
|
|
|
|
|
|
|
2. No [WAL](https://en.wikipedia.org/wiki/Write-ahead_logging) means
|
|
|
|
relatively big [WAF](https://en.wikipedia.org/wiki/Write_amplification)
|
|
|
|
(Write Amplification Factor). Because of this syncing data to disk might
|
|
|
|
be quite resource intensive and be main performance bottleneck during
|
|
|
|
intensive write workload.
|
|
|
|
> As compromise _libmdbx_ allows several modes of lazy and/or periodic
|
|
|
|
> syncing, including `MAPASYNC` mode, which modificate data in memory and
|
|
|
|
> asynchronously syncs data to disk, moment to sync is picked by OS.
|
|
|
|
>
|
|
|
|
> Although this should be used with care, synchronous transactions in a DB
|
|
|
|
> with transaction journal will require 2 IOPS minimum (probably 3-4 in
|
|
|
|
> practice) because of filesystem overhead, overhead depends on
|
|
|
|
> filesystem, not on record count or record size. In _libmdbx_ IOPS count
|
|
|
|
> will grow logarithmically depending on record count in DB (height of B+
|
|
|
|
> tree) and will require at least 2 IOPS per transaction too.
|
|
|
|
|
|
|
|
3. [CoW](https://en.wikipedia.org/wiki/Copy-on-write) for
|
|
|
|
[MVCC](https://en.wikipedia.org/wiki/Multiversion_concurrency_control)
|
|
|
|
is done on memory page level with
|
|
|
|
[B+trees](https://ru.wikipedia.org/wiki/B-%D0%B4%D0%B5%D1%80%D0%B5%D0%B2%D0%BE).
|
|
|
|
Therefore altering data requires to copy about Olog(N) memory pages,
|
|
|
|
which uses [memory bandwidth](https://en.wikipedia.org/wiki/Memory_bandwidth) and is main
|
|
|
|
performance bottleneck in `MAPASYNC` mode.
|
|
|
|
> This is unavoidable, but isn't that bad. Syncing data to disk requires
|
|
|
|
> much more similar operations which will be done by OS, therefore this is
|
|
|
|
> noticeable only if data sync to persistent storage is fully disabled.
|
|
|
|
> _libmdbx_ allows to safely save data to persistent storage with minimal
|
|
|
|
> performance overhead. If there is no need to save data to persistent
|
|
|
|
> storage then it's much more preferable to use `std::map`.
|
|
|
|
|
|
|
|
|
|
|
|
4. LMDB has a problem of long-time readers which degrades performance
|
|
|
|
and bloats DB.
|
|
|
|
> _libmdbx_ addresses that, details below.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
5. _LMDB_ is susceptible to DB corruption in `WRITEMAP+MAPASYNC` mode.
|
|
|
|
_libmdbx_ in `WRITEMAP+MAPASYNC` guarantees DB integrity and consistency
|
|
|
|
of data.
|
|
|
|
> Additionally there is an alternative: `UTTERLY_NOSYNC` mode.
|
|
|
|
> Details below.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
#### Long-time read transactions problem
|
|
|
|
Garbage collection problem exists in all databases one way or another
|
|
|
|
(e.g. VACUUM in PostgreSQL). But in _libmdbx_ and LMDB it's even more
|
|
|
|
important because of high performance and deliberate simplification of
|
|
|
|
internals with emphasis on performance.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* Altering data during long read operation may exhaust available space
|
|
|
|
on persistent storage.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* If available space is exhausted then any attempt to update data
|
|
|
|
results in `MAP_FULL` error until long read operation ends.
|
2017-11-15 23:47:39 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* Main examples of long readers is hot backup and debugging of client
|
|
|
|
application which actively uses read transactions.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* In _LMDB_ this results in degraded performance of all operations of
|
|
|
|
syncing data to persistent storage.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* _libmdbx_ has a mechanism which aborts such operations and `LIFO RECLAIM`
|
|
|
|
mode which addresses performance degradation.
|
|
|
|
|
|
|
|
Read operations operate only over snapshot of DB which is consistent on
|
|
|
|
the moment when read transaction started. This snapshot doesn't change
|
|
|
|
throughout the transaction but this leads to inability to reclaim the
|
|
|
|
pages until read transaction ends.
|
|
|
|
|
|
|
|
In _LMDB_ this leads to a problem that memory pages, allocated for
|
|
|
|
operations during long read, will be used for operations and won't be
|
|
|
|
reclaimed until DB process terminates. In _LMDB_ they are used in
|
|
|
|
[FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))
|
|
|
|
manner, which causes increased page count and less chance of cache hit
|
|
|
|
during I/O. In other words: one long-time reader can impact performance
|
|
|
|
of all database until it'll be reopened.
|
|
|
|
|
|
|
|
_libmdbx_ addresses the problem, details below. Illustrations to this
|
|
|
|
problem can be found in the
|
|
|
|
[presentation](http://www.slideshare.net/leoyuriev/lmdb). There is also
|
|
|
|
example of performance increase thanks to
|
|
|
|
[BBWC](https://en.wikipedia.org/wiki/Disk_buffer#Write_acceleration)
|
|
|
|
when `LIFO RECLAIM` enabled in _libmdbx_.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
#### Data safety in async-write mode
|
|
|
|
In `WRITEMAP+MAPSYNC` mode dirty pages are written to persistent storage
|
|
|
|
by kernel. This means that in case of application crash OS kernel will
|
|
|
|
write all dirty data to disk and nothing will be lost. But in case of
|
|
|
|
hardware malfunction or OS kernel fatal error only some dirty data might
|
|
|
|
be synced to disk, and there is high probability that pages with
|
|
|
|
metadata saved, will point to non-saved, hence non-existent, data pages.
|
|
|
|
In such situation, DB is completely corrupted and can't be repaired even
|
|
|
|
if there was full sync before the crash via `mdbx_env_sync().
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
_libmdbx_ addresses this by fully reimplementing write path of data:
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* In `WRITEMAP+MAPSYNC` mode meta-data pages aren't updated in place,
|
|
|
|
instead their shadow copies are used and their updates are synced after
|
|
|
|
data is flushed to disk.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* During transaction commit _libmdbx_ marks synchronization points as
|
|
|
|
steady or weak depending on how much synchronization needed between RAM
|
|
|
|
and persistent storage, e.g. in `WRITEMAP+MAPSYNC` commited transactions
|
|
|
|
are marked as weak, but during explicit data synchronization - as
|
|
|
|
steady.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* _libmdbx_ maintains three separate meta-pages instead of two. This
|
|
|
|
allows to commit transaction with steady or weak synchronization point
|
|
|
|
without losing two previous synchronization points (one of them can be
|
|
|
|
steady, and second - weak). This allows to order weak and steady
|
|
|
|
synchronization points in any order without losing consistency in case
|
|
|
|
of system crash.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
* During DB open _libmdbx_ rollbacks to the last steady synchronization
|
|
|
|
point, this guarantees database integrity.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
For data safety pages which form database snapshot with steady
|
|
|
|
synchronization point must not be updated until next steady
|
|
|
|
synchronization point. So last steady synchronization point creates
|
|
|
|
"long-time read" effect. The only difference that in case of memory
|
|
|
|
exhaustion the problem will be immediately addressed by flushing changes
|
|
|
|
to persistent storage and forming new steady synchronization point.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
So in async-write mode _libmdbx_ will always use new pages until memory
|
|
|
|
is exhausted or `mdbx_env_sync()` is invoked. Total disk usage will be
|
|
|
|
almost the same as in sync-write mode.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
Current _libmdbx_ gives a choice of safe async-write mode (default) and
|
|
|
|
`UTTERLY_NOSYNC` mode which may result in full DB corruption during
|
|
|
|
system crash as with LMDB.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
Next version of _libmdbx_ will create steady synchronization points
|
|
|
|
automatically in async-write mode.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
--------------------------------------------------------------------------------
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
Performance comparison
|
|
|
|
======================
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
All benchmarks were done by [IOArena](https://github.com/pmwkaa/ioarena)
|
|
|
|
and multiple [scripts](https://github.com/pmwkaa/ioarena/tree/HL%2B%2B2015)
|
|
|
|
runs on Lenovo Carbon-2 laptop, i7-4600U 2.1 GHz, 8 Gb RAM,
|
|
|
|
SSD SAMSUNG MZNTD512HAGL-000L1 (DXT23L0Q) 512 Gb.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
--------------------------------------------------------------------------------
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
### Integral performance
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
Here showed sum of performance metrics in 3 benchmarks:
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Read/Search on 4 CPU cores machine;
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Transactions with [CRUD](https://en.wikipedia.org/wiki/CRUD)
|
|
|
|
operations in sync-write mode (fdatasync is called after each
|
|
|
|
transaction);
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Transactions with [CRUD](https://en.wikipedia.org/wiki/CRUD)
|
|
|
|
operations in lazy-write mode (moment to sync data to persistent storage
|
|
|
|
is decided by OS).
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
*Reasons why asynchronous mode isn't benchmarked here:*
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
1. It doesn't make sense as it has to be done with DB engines, oriented
|
|
|
|
for keeping data in memory e.g. [Tarantool](https://tarantool.io/),
|
|
|
|
[Redis](https://redis.io/)), etc.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
2. Performance gap is too high to compare in any meaningful way.
|
2018-01-18 03:19:59 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
![Comparison #1: Integral Performance](https://raw.githubusercontent.com/wiki/leo-yuriev/libmdbx/img/perf-slide-1.png)
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2017-07-04 17:07:05 +08:00
|
|
|
--------------------------------------------------------------------------------
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
### Read Scalability
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
Summary performance with concurrent read/search queries in 1-2-4-8
|
|
|
|
threads on 4 CPU cores machine.
|
2017-01-09 06:19:24 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
![Comparison #2: Read Scalability](https://raw.githubusercontent.com/wiki/leo-yuriev/libmdbx/img/perf-slide-2.png)
|
2017-01-09 06:19:24 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
--------------------------------------------------------------------------------
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
### Sync-write mode
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Linear scale on left and dark rectangles mean arithmetic mean
|
|
|
|
transactions per second;
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Logarithmic scale on right is in seconds and yellow intervals mean
|
|
|
|
execution time of transactions. Each interval shows minimal and maximum
|
|
|
|
execution time, cross marks standard deviation.
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
**10,000 transactions in sync-write mode**. In case of a crash all data
|
|
|
|
is consistent and state is right after last successful transaction.
|
|
|
|
[fdatasync](https://linux.die.net/man/2/fdatasync) syscall is used after
|
|
|
|
each write transaction in this mode.
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
In the benchmark each transaction contains combined CRUD operations (2
|
|
|
|
inserts, 1 read, 1 update, 1 delete). Benchmark starts on empty database
|
|
|
|
and after full run the database contains 10,000 small key-value records.
|
2017-01-18 21:40:56 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
![Comparison #3: Sync-write mode](https://raw.githubusercontent.com/wiki/leo-yuriev/libmdbx/img/perf-slide-3.png)
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
--------------------------------------------------------------------------------
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
### Lazy-write mode
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Linear scale on left and dark rectangles mean arithmetic mean of
|
|
|
|
thousands transactions per second;
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Logarithmic scale on right in seconds and yellow intervals mean
|
|
|
|
execution time of transactions. Each interval shows minimal and maximum
|
|
|
|
execution time, cross marks standard deviation.
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
**100,000 transactions in lazy-write mode**. In case of a crash all data
|
|
|
|
is consistent and state is right after one of last transactions, but
|
|
|
|
transactions after it will be lost. Other DB engines use
|
|
|
|
[WAL](https://en.wikipedia.org/wiki/Write-ahead_logging) or transaction
|
|
|
|
journal for that, which in turn depends on order of operations in
|
|
|
|
journaled filesystem. _libmdbx_ doesn't use WAL and hands I/O operations
|
|
|
|
to filesystem and OS kernel (mmap).
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
In the benchmark each transaction contains combined CRUD operations (2
|
|
|
|
inserts, 1 read, 1 update, 1 delete). Benchmark starts on empty database
|
|
|
|
and after full run the database contains 100,000 small key-value
|
|
|
|
records.
|
2017-01-09 02:48:00 +08:00
|
|
|
|
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
![Comparison #4: Lazy-write mode](https://raw.githubusercontent.com/wiki/leo-yuriev/libmdbx/img/perf-slide-4.png)
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
--------------------------------------------------------------------------------
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
### Async-write mode
|
2017-01-09 02:48:00 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Linear scale on left and dark rectangles mean arithmetic mean of
|
|
|
|
thousands transactions per second;
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Logarithmic scale on right in seconds and yellow intervals mean
|
|
|
|
execution time of transactions. Each interval shows minimal and maximum
|
|
|
|
execution time, cross marks standard deviation.
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
**1,000,000 transactions in async-write mode**. In case of a crash all
|
|
|
|
data will be consistent and state will be right after one of last
|
|
|
|
transactions, but lost transaction count is much higher than in
|
|
|
|
lazy-write mode. All DB engines in this mode do as little writes as
|
|
|
|
possible on persistent storage. _libmdbx_ uses
|
|
|
|
[msync(MS_ASYNC)](https://linux.die.net/man/2/msync) in this mode.
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
In the benchmark each transaction contains combined CRUD operations (2
|
|
|
|
inserts, 1 read, 1 update, 1 delete). Benchmark starts on empty database
|
|
|
|
and after full run the database contains 10,000 small key-value records.
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
![Comparison #5: Async-write mode](https://raw.githubusercontent.com/wiki/leo-yuriev/libmdbx/img/perf-slide-5.png)
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
--------------------------------------------------------------------------------
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
### Cost comparison
|
2017-02-08 22:52:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
Summary of used resources during lazy-write mode benchmarks:
|
2017-02-18 03:02:07 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Read and write IOPS;
|
2017-02-18 03:02:07 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Sum of user CPU time and sys CPU time;
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
- Used space on persistent storage after the test and closed DB, but not
|
|
|
|
waiting for the end of all internal housekeeping operations (LSM
|
|
|
|
compactification, etc).
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
_ForestDB_ is excluded because benchmark showed it's resource
|
|
|
|
consumption for each resource (CPU, IOPS) much higher than other engines
|
|
|
|
which prevents to meaningfully compare it with them.
|
2017-07-04 17:07:05 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
All benchmark data is gathered by
|
|
|
|
[getrusage()](http://man7.org/linux/man-pages/man2/getrusage.2.html)
|
|
|
|
syscall and by scanning data directory.
|
2017-07-27 23:16:19 +08:00
|
|
|
|
2018-07-30 07:55:13 +08:00
|
|
|
![Comparison #6: Cost comparison](https://raw.githubusercontent.com/wiki/leo-yuriev/libmdbx/img/perf-slide-6.png)
|
2017-07-27 23:38:08 +08:00
|
|
|
|
2017-07-27 23:16:19 +08:00
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
|
|
|
|
```
|
|
|
|
$ objdump -f -h -j .text libmdbx.so
|
|
|
|
|
|
|
|
libmdbx.so: file format elf64-x86-64
|
|
|
|
architecture: i386:x86-64, flags 0x00000150:
|
|
|
|
HAS_SYMS, DYNAMIC, D_PAGED
|
|
|
|
start address 0x000030e0
|
|
|
|
|
|
|
|
Sections:
|
2018-01-08 23:49:21 +08:00
|
|
|
Idx Name Size VMA LMA File off Algn
|
|
|
|
11 .text 00014d84 00000000000030e0 00000000000030e0 000030e0 2**4
|
2017-07-27 23:16:19 +08:00
|
|
|
CONTENTS, ALLOC, LOAD, READONLY, CODE
|
2018-01-08 23:49:21 +08:00
|
|
|
|
2017-07-27 23:16:19 +08:00
|
|
|
```
|