ISAR/libmdbx
3
0
Fork 0
mirror of https://github.com/isar/libmdbx.git synced 2025-01-30 22:47:16 +08:00
Code Issues Packages Projects Releases Wiki Activity

mdbx: update ChangeLog and captions.

Browse Source
This commit is contained in:
Леонид Юрьев (Leonid Yuriev) 2022-04-20 23:10:00 +03:00
parent 35b5abc103
commit 9aa53d1616
3 changed files with 120 additions and 82 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
ChangeLog.md
View File

@ -3,21 +3,29 @@ ChangeLog
### TODO
- [Engage an "overlapped I/O" on Windows](https://erased_by_erased_by_github.none/libmdbx/issues/224).
- [Simple careful mode for working with corrupted DB](https://erased_by_erased_by_github.none/libmdbx/issues/223).
- [Move most of `mdbx_chk` functional to the library API](https://erased_by_erased_by_github.none/libmdbx/issues/204).
- [Replace SRW-lock on Windows to allow shrink DB with `MDBX_NOTLS` option](https://erased_by_erased_by_github.none/libmdbx/issues/210).
- [More flexible support of asynchronous runtime/framework(s)](https://erased_by_erased_by_github.none/libmdbx/issues/200).
- [Migration guide from LMDB to MDBX](https://erased_by_erased_by_github.none/libmdbx/issues/199).
- [Get rid of dirty-pages list in MDBX_WRITEMAP mode](https://erased_by_erased_by_github.none/libmdbx/issues/193).
- [Large/Overflow pages accounting for dirty-room](https://erased_by_erased_by_github.none/libmdbx/issues/192).
- [Support for RAW devices](https://erased_by_erased_by_github.none/libmdbx/issues/124).
- [Support MessagePack for Keys & Values](https://erased_by_erased_by_github.none/libmdbx/issues/115).
- [Engage new terminology](https://erased_by_erased_by_github.none/libmdbx/issues/137).
- [Engage an "overlapped I/O" on Windows](https://erased_by_github.none/libmdbx/issues/224).
- [Simple careful mode for working with corrupted DB](https://erased_by_github.none/libmdbx/issues/223).
- [Move most of `mdbx_chk` functional to the library API](https://erased_by_github.none/libmdbx/issues/204).
- [Replace SRW-lock on Windows to allow shrink DB with `MDBX_NOTLS` option](https://erased_by_github.none/libmdbx/issues/210).
- [More flexible support of asynchronous runtime/framework(s)](https://erased_by_github.none/libmdbx/issues/200).
- [Migration guide from LMDB to MDBX](https://erased_by_github.none/libmdbx/issues/199).
- [Get rid of dirty-pages list in MDBX_WRITEMAP mode](https://erased_by_github.none/libmdbx/issues/193).
- [Large/Overflow pages accounting for dirty-room](https://erased_by_github.none/libmdbx/issues/192).
- [Support for RAW devices](https://erased_by_github.none/libmdbx/issues/124).
- [Support MessagePack for Keys & Values](https://erased_by_github.none/libmdbx/issues/115).
- [Engage new terminology](https://erased_by_github.none/libmdbx/issues/137).
- Packages for [Astra Linux](https://astralinux.ru/), [ALT Linux](https://www.altlinux.org/), [ROSA Linux](https://www.rosalinux.ru/), etc.
## v0.11.7 (underway)
## v0.11.7 (scheduled at 2022-04-22)
Stable release recovery after the Github's intentional malicious disaster.
The origin repository of the project has been migrated
to [GitFlic](https://gitflic.ru/project/erthink/libmdbx) since on April 15, 2022,
the Github administration, without any warning not explanation, deleted
libmdbx along with a lot of other projects, simultaneously blocking
access to many developers. For the same reason ~~Github~~ is blacklisted forever.
New:
@ -44,7 +52,7 @@ Minors:
- Resolved all warnings from MinGW while used without CMake.
## v0.11.6 (scheduled for 2022-03-24)
## v0.11.6 at 2022-03-24
The stable release with the complete workaround for an incoherence flaw of Linux unified page/buffer cache.
Nonetheless the cause for this trouble may be an issue of Intel CPU cache/MESI.

79
README.md
View File

@ -1,32 +1,47 @@
<!-- Required extensions: pymdownx.betterem, pymdownx.tilde, pymdownx.emoji, pymdownx.tasklist, pymdownx.superfences -->
### The origin repository of the project has been moved to https://gitflic.ru/project/erthink/libmdbx since on April 15, 2022, the Github administration, without warning and without explanation, deleted libmdbx along with a lot of other projects, simultaneously blocking access to many developers.
For the same reason Github is blacklisted forever.
### The origin repository of the project has been migrated
to [GitFlic](https://gitflic.ru/project/erthink/libmdbx) since on April 15, 2022,
the Github administration, without any warning not explanation, deleted
libmdbx along with a lot of other projects, simultaneously blocking
access to many developers.
### Основной репозиторий проекта перемещен на https://gitflic.ru/project/erthink/libmdbx, так как 15 апреля 2022 администрация Github без предупреждения и без объяснения причин удалила libmdbx вместе с массой других проектов, одновременно заблокировав доступ многим разработчикам.
По той же причине Github навсегда занесен в черный список.
For the same reason ~~Github~~ is blacklisted forever.
### Основной репозиторий проекта перемещен
на [GitFlic](https://gitflic.ru/project/erthink/libmdbx), так как 15 апреля 2022
администрация Github без предупреждения и объяснения причин удалила
libmdbx вместе с массой других проектов, одновременно заблокировав
доступ многим разработчикам.
По этой же причине ~~Github~~ навсегда занесен в черный список.
--------------------------------------------------------------------------------
*The Future will (be) [Positive](https://www.ptsecurity.com). Всё будет хорошо.*
> Please refer to the online [documentation](https://libmdbx.website.yandexcloud.net)
> with [`C` API description](https://libmdbx.website.yandexcloud.net/group__c__api.html)
> and pay attention to the [`C++` API](https://gitflic.ru/project/erthink/libmdbx/blob?file=mdbx.h%2B%2B#line-num-1).
> Questions, feedback and suggestions are welcome to the [Telegram' group](https://t.me/libmdbx).
> For NEWS take a look to the [ChangeLog](./ChangeLog.md).
> For NEWS take a look to the [ChangeLog](https://gitflic.ru/project/erthink/libmdbx/blob?file=ChangeLog.md).
libmdbx
========
<!-- section-begin overview -->
_libmdbx_ is an extremely fast, compact, powerful, embedded,
transactional [key-value database](https://en.wikipedia.org/wiki/Key-value_database),
with [permissive license](./LICENSE).
_libmdbx_ is an extremely fast, compact, powerful, embedded, transactional
[key-value database](https://en.wikipedia.org/wiki/Key-value_database),
with [permissive license](https://gitflic.ru/project/erthink/libmdbx/blob?file=LICENSE).
_libmdbx_ has a specific set of properties and capabilities,
focused on creating unique lightweight solutions.
1. Allows **a swarm of multi-threaded processes to
[ACID]((https://en.wikipedia.org/wiki/ACID))ly read and update** several
[ACID](https://en.wikipedia.org/wiki/ACID)ly read and update** several
key-value [maps](https://en.wikipedia.org/wiki/Associative_array) and
[multimaps](https://en.wikipedia.org/wiki/Multimap) in a locally-shared
database.
@ -59,6 +74,7 @@ neglected in favour of write performance.
7. Supports Linux, Windows, MacOS, Android, iOS, FreeBSD, DragonFly, Solaris,
OpenSolaris, OpenIndiana, NetBSD, OpenBSD and other systems compliant with
**POSIX.1-2008**.
<!-- section-end -->
Historically, _libmdbx_ is a deeply revised and extended descendant of the amazing
@ -66,16 +82,19 @@ Historically, _libmdbx_ is a deeply revised and extended descendant of the amazi
_libmdbx_ inherits all benefits from _LMDB_, but resolves some issues and adds [a set of improvements](#improvements-beyond-lmdb).
<!-- section-begin mithril -->
The next version is under active non-public development from scratch and will be
released as _**MithrilDB**_ and `libmithrildb` for libraries & packages.
released as **MithrilDB** and `libmithrildb` for libraries & packages.
Admittedly mythical [Mithril](https://en.wikipedia.org/wiki/Mithril) is
resembling silver but being stronger and lighter than steel. Therefore
_MithrilDB_ is a rightly relevant name.
> _MithrilDB_ will be radically different from _libmdbx_ by the new
> database format and API based on C++17, as well as the [Apache 2.0
> License](https://www.apache.org/licenses/LICENSE-2.0). The goal of this
> revolution is to provide a clearer and robust API, add more features and
> new valuable properties of the database.
<!-- section-end -->
-----
@ -185,6 +204,7 @@ databases"](https://github.com/coreos/bbolt#comparison-with-other-databases)
which is also (mostly) applicable to _libmdbx_.
<!-- section-end -->
<!-- section-begin improvements -->
Improvements beyond LMDB
@ -200,11 +220,13 @@ the user's point of view.
## Added Features
1. Keys could be more than 2 times longer than _LMDB_.
> For DB with default page size _libmdbx_ support keys up to 2022 bytes
> and up to 32742 bytes for 64K page size. _LMDB_ allows key size up to
> 511 bytes and may silently loses data with large values.
2. Up to 30% faster than _LMDB_ in [CRUD](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete) benchmarks.
> Benchmarks of the in-[tmpfs](https://en.wikipedia.org/wiki/Tmpfs) scenarios,
> that tests the speed of the engine itself, showned that _libmdbx_ 10-20% faster than _LMDB_,
> and up to 30% faster when _libmdbx_ compiled with specific build options
@ -215,6 +237,7 @@ the user's point of view.
> and [WiredTiger](https://en.wikipedia.org/wiki/WiredTiger).
3. Automatic on-the-fly database size adjustment, both increment and reduction.
> _libmdbx_ manages the database size according to parameters specified
> by `mdbx_env_set_geometry()` function,
> ones include the growth step and the truncation threshold.
@ -223,13 +246,16 @@ the user's point of view.
> due to its internal limitations and unimplemented functions, i.e. the `MDBX_UNABLE_EXTEND_MAPSIZE` error will be returned.
4. Automatic continuous zero-overhead database compactification.
> During each commit _libmdbx_ merges a freeing pages which adjacent with the unallocated area
> at the end of file, and then truncates unused space when a lot enough of.
5. The same database format for 32- and 64-bit builds.
> _libmdbx_ database format depends only on the [endianness](https://en.wikipedia.org/wiki/Endianness) but not on the [bitness](https://en.wiktionary.org/wiki/bitness).
6. LIFO policy for Garbage Collection recycling. This can significantly increase write performance due write-back disk cache up to several times in a best case scenario.
> LIFO means that for reuse will be taken the latest becomes unused pages.
> Therefore the loop of database pages circulation becomes as short as possible.
> In other words, the set of pages, that are (over)written in memory and on disk during a series of write transactions, will be as small as possible.
@ -238,6 +264,7 @@ the user's point of view.
7. Fast estimation of range query result volume, i.e. how many items can
be found between a `KEY1` and a `KEY2`. This is a prerequisite for build
and/or optimize query execution plans.
> _libmdbx_ performs a rough estimate based on common B-tree pages of the paths from root to corresponding keys.
8. `mdbx_chk` utility for database integrity check.
@ -251,10 +278,12 @@ Since version 0.9.1, the utility supports checking the database using any of the
or not, that allows to avoid copy-out before updates.
12. Extended information of whole-database, sub-databases, transactions, readers enumeration.
> _libmdbx_ provides a lot of information, including dirty and leftover pages
> for a write transaction, reading lag and holdover space for read transactions.
13. Extended update and delete operations.
> _libmdbx_ allows one _at once_ with getting previous value
> and addressing the particular item from multi-value with the same key.
@ -286,6 +315,7 @@ pre-opening is not needed.
4. Returning `MDBX_EMULTIVAL` error in case of ambiguous update or delete.
5. Guarantee of database integrity even in asynchronous unordered write-to-disk mode.
> _libmdbx_ propose additional trade-off by `MDBX_SAFE_NOSYNC` with append-like manner for updates,
> that avoids database corruption after a system crash contrary to LMDB.
> Nevertheless, the `MDBX_UTTERLY_NOSYNC` mode is available to match LMDB's behaviour for `MDB_NOSYNC`.
@ -320,7 +350,7 @@ year later _libmdbx_ was separated into a standalone project, which was
[presented at Highload++ 2015
conference](http://www.highload.ru/2015/abstracts/1831.html).
Since 2017 _libmdbx_ is used in [Fast Positive Tables](https://erased_by_github.none/libfpta),
Since 2017 _libmdbx_ is used in [Fast Positive Tables](https://gitflic.ru/project/erthink/libfpta),
and development is funded by [Positive Technologies](https://www.ptsecurity.com).
## Acknowledgments
@ -338,27 +368,23 @@ Usage
=====
<!-- section-begin usage -->
Currently, libmdbx is only available in a
[source code](https://en.wikipedia.org/wiki/Source_code) form.
Packages support for common Linux distributions is planned in the future,
since release the version 1.0.
## Never use tarballs nor zips automatically provided by Github !
Please don't use tarballs nor zips which are automatically provided by Github.
These archives do not contain version information and thus are unfit to build _libmdbx_.
Instead of ones just clone the git repository, either download a tarball or zip with the properly amalgamated source core.
Moreover, please vote for [ability of disabling auto-creation such unsuitable archives](https://github.community/t/disable-tarball).
## Source code embedding
_libmdbx_ provides two official ways for integration in source code form:
1. Using the amalgamated source code.
> The amalgamated source code includes all files required to build and
> use _libmdbx_, but not for testing _libmdbx_ itself.
2. Adding the complete original source code as a `git submodule`.
> This allows you to build as _libmdbx_ and testing tool.
> On the other hand, this way requires you to pull git tags, and use C++11 compiler for test tool.
@ -387,8 +413,7 @@ and build options respectively.
The only significant specificity is that git' tags are required
to build from complete (not amalgamated) source codes.
Executing **`git fetch --tags --force --prune`** is enough to get ones,
or `git fetch --unshallow --tags --prune --force` after the Github's
[`actions/checkout@v2`](https://github.com/actions/checkout) either set **`fetch-depth: 0`** for it.
and `--unshallow` or `--update-shallow` is required for shallow cloned case.
So just using CMake or GNU Make in your habitual manner and feel free to
fill an issue or make pull request in the case something will be
@ -558,7 +583,7 @@ from the [ios-cmake](https://github.com/leetal/ios-cmake) project.
## API description
Please refer to the online [_libmdbx_ API reference](https://erased_by_github_and_to_be_restored.todo/libmdbx/)
Please refer to the online [_libmdbx_ API reference](https://libmdbx.website.yandexcloud.net/docs)
and/or see the [mdbx.h++](mdbx.h%2B%2B) and [mdbx.h](mdbx.h) headers.
<!-- section-begin bindings -->
@ -617,7 +642,7 @@ Here showed sum of performance metrics in 3 benchmarks:
2. Performance gap is too high to compare in any meaningful way.
![Comparison #1: Integral Performance](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-1.png)
![Comparison #1: Integral Performance](https://libmdbx.website.yandexcloud.net/img/perf-slide-1.png)
--------------------------------------------------------------------------------
@ -626,7 +651,7 @@ Here showed sum of performance metrics in 3 benchmarks:
Summary performance with concurrent read/search queries in 1-2-4-8
threads on the machine with 4 logical CPUs in HyperThreading mode (i.e. actually 2 physical CPU cores).
![Comparison #2: Read Scalability](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-2.png)
![Comparison #2: Read Scalability](https://libmdbx.website.yandexcloud.net/img/perf-slide-2.png)
--------------------------------------------------------------------------------
@ -648,7 +673,7 @@ In the benchmark each transaction contains combined CRUD operations (2
inserts, 1 read, 1 update, 1 delete). Benchmark starts on an empty database
and after full run the database contains 10,000 small key-value records.
![Comparison #3: Sync-write mode](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-3.png)
![Comparison #3: Sync-write mode](https://libmdbx.website.yandexcloud.net/img/perf-slide-3.png)
--------------------------------------------------------------------------------
@ -675,7 +700,7 @@ and after full run the database contains 100,000 small key-value
records.
![Comparison #4: Lazy-write mode](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-4.png)
![Comparison #4: Lazy-write mode](https://libmdbx.website.yandexcloud.net/img/perf-slide-4.png)
--------------------------------------------------------------------------------
@ -699,7 +724,7 @@ In the benchmark each transaction contains combined CRUD operations (2
inserts, 1 read, 1 update, 1 delete). Benchmark starts on an empty database
and after full run the database contains 10,000 small key-value records.
![Comparison #5: Async-write mode](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-5.png)
![Comparison #5: Async-write mode](https://libmdbx.website.yandexcloud.net/img/perf-slide-5.png)
--------------------------------------------------------------------------------
@ -723,6 +748,6 @@ All benchmark data is gathered by
[getrusage()](http://man7.org/linux/man-pages/man2/getrusage.2.html)
syscall and by scanning the data directory.
![Comparison #6: Cost comparison](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-6.png)
![Comparison #6: Cost comparison](https://libmdbx.website.yandexcloud.net/img/perf-slide-6.png)
<!-- section-end -->

7
docs/_toc.md
View File

@ -1,3 +1,8 @@
The origin repository of the project has been migrated
to [GitFlic](https://gitflic.ru/project/erthink/libmdbx) since on April 15, 2022,
the Github administration, without any warning not explanation, deleted
libmdbx along with a lot of other projects, simultaneously blocking
access to many developers. For the same reason ~~Github~~ is blacklisted forever.
_The Future will (be) [Positive](https://www.ptsecurity.com). Всё будет хорошо._
@ -23,7 +28,7 @@ each of which is divided into several sections.
- The \ref mdbx.h++ header file reference
Please do not hesitate to point out errors in the documentation,
including creating [PR](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/proposing-changes-to-your-work-with-pull-requests) with corrections and improvements.
including creating [merge-request](https://gitflic.ru/project/erthink/libmdbx/merge-request) with corrections and improvements.
---