diff --git a/ChangeLog.md b/ChangeLog.md index e2c3a8fd..0805264b 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -3,21 +3,29 @@ ChangeLog ### TODO - - [Engage an "overlapped I/O" on Windows](https://erased_by_erased_by_github.none/libmdbx/issues/224). - - [Simple careful mode for working with corrupted DB](https://erased_by_erased_by_github.none/libmdbx/issues/223). - - [Move most of `mdbx_chk` functional to the library API](https://erased_by_erased_by_github.none/libmdbx/issues/204). - - [Replace SRW-lock on Windows to allow shrink DB with `MDBX_NOTLS` option](https://erased_by_erased_by_github.none/libmdbx/issues/210). - - [More flexible support of asynchronous runtime/framework(s)](https://erased_by_erased_by_github.none/libmdbx/issues/200). - - [Migration guide from LMDB to MDBX](https://erased_by_erased_by_github.none/libmdbx/issues/199). - - [Get rid of dirty-pages list in MDBX_WRITEMAP mode](https://erased_by_erased_by_github.none/libmdbx/issues/193). - - [Large/Overflow pages accounting for dirty-room](https://erased_by_erased_by_github.none/libmdbx/issues/192). - - [Support for RAW devices](https://erased_by_erased_by_github.none/libmdbx/issues/124). - - [Support MessagePack for Keys & Values](https://erased_by_erased_by_github.none/libmdbx/issues/115). - - [Engage new terminology](https://erased_by_erased_by_github.none/libmdbx/issues/137). + - [Engage an "overlapped I/O" on Windows](https://erased_by_github.none/libmdbx/issues/224). + - [Simple careful mode for working with corrupted DB](https://erased_by_github.none/libmdbx/issues/223). + - [Move most of `mdbx_chk` functional to the library API](https://erased_by_github.none/libmdbx/issues/204). + - [Replace SRW-lock on Windows to allow shrink DB with `MDBX_NOTLS` option](https://erased_by_github.none/libmdbx/issues/210). + - [More flexible support of asynchronous runtime/framework(s)](https://erased_by_github.none/libmdbx/issues/200). + - [Migration guide from LMDB to MDBX](https://erased_by_github.none/libmdbx/issues/199). + - [Get rid of dirty-pages list in MDBX_WRITEMAP mode](https://erased_by_github.none/libmdbx/issues/193). + - [Large/Overflow pages accounting for dirty-room](https://erased_by_github.none/libmdbx/issues/192). + - [Support for RAW devices](https://erased_by_github.none/libmdbx/issues/124). + - [Support MessagePack for Keys & Values](https://erased_by_github.none/libmdbx/issues/115). + - [Engage new terminology](https://erased_by_github.none/libmdbx/issues/137). - Packages for [Astra Linux](https://astralinux.ru/), [ALT Linux](https://www.altlinux.org/), [ROSA Linux](https://www.rosalinux.ru/), etc. -## v0.11.7 (underway) +## v0.11.7 (scheduled at 2022-04-22) + +Stable release recovery after the Github's intentional malicious disaster. + +The origin repository of the project has been migrated +to [GitFlic](https://gitflic.ru/project/erthink/libmdbx) since on April 15, 2022, +the Github administration, without any warning not explanation, deleted +libmdbx along with a lot of other projects, simultaneously blocking +access to many developers. For the same reason ~~Github~~ is blacklisted forever. New: @@ -44,7 +52,7 @@ Minors: - Resolved all warnings from MinGW while used without CMake. -## v0.11.6 (scheduled for 2022-03-24) +## v0.11.6 at 2022-03-24 The stable release with the complete workaround for an incoherence flaw of Linux unified page/buffer cache. Nonetheless the cause for this trouble may be an issue of Intel CPU cache/MESI. diff --git a/README.md b/README.md index df97abaa..5c03ce3e 100644 --- a/README.md +++ b/README.md @@ -1,32 +1,47 @@ -### The origin repository of the project has been moved to https://gitflic.ru/project/erthink/libmdbx since on April 15, 2022, the Github administration, without warning and without explanation, deleted libmdbx along with a lot of other projects, simultaneously blocking access to many developers. -For the same reason Github is blacklisted forever. +### The origin repository of the project has been migrated +to [GitFlic](https://gitflic.ru/project/erthink/libmdbx) since on April 15, 2022, +the Github administration, without any warning not explanation, deleted +libmdbx along with a lot of other projects, simultaneously blocking +access to many developers. -### Основной репозиторий проекта перемещен на https://gitflic.ru/project/erthink/libmdbx, так как 15 апреля 2022 администрация Github без предупреждения и без объяснения причин удалила libmdbx вместе с массой других проектов, одновременно заблокировав доступ многим разработчикам. -По той же причине Github навсегда занесен в черный список. +For the same reason ~~Github~~ is blacklisted forever. + +### Основной репозиторий проекта перемещен +на [GitFlic](https://gitflic.ru/project/erthink/libmdbx), так как 15 апреля 2022 +администрация Github без предупреждения и объяснения причин удалила +libmdbx вместе с массой других проектов, одновременно заблокировав +доступ многим разработчикам. + +По этой же причине ~~Github~~ навсегда занесен в черный список. -------------------------------------------------------------------------------- *The Future will (be) [Positive](https://www.ptsecurity.com). Всё будет хорошо.* +> Please refer to the online [documentation](https://libmdbx.website.yandexcloud.net) +> with [`C` API description](https://libmdbx.website.yandexcloud.net/group__c__api.html) +> and pay attention to the [`C++` API](https://gitflic.ru/project/erthink/libmdbx/blob?file=mdbx.h%2B%2B#line-num-1). + > Questions, feedback and suggestions are welcome to the [Telegram' group](https://t.me/libmdbx). -> For NEWS take a look to the [ChangeLog](./ChangeLog.md). +> For NEWS take a look to the [ChangeLog](https://gitflic.ru/project/erthink/libmdbx/blob?file=ChangeLog.md). libmdbx ======== -_libmdbx_ is an extremely fast, compact, powerful, embedded, -transactional [key-value database](https://en.wikipedia.org/wiki/Key-value_database), -with [permissive license](./LICENSE). + +_libmdbx_ is an extremely fast, compact, powerful, embedded, transactional +[key-value database](https://en.wikipedia.org/wiki/Key-value_database), +with [permissive license](https://gitflic.ru/project/erthink/libmdbx/blob?file=LICENSE). _libmdbx_ has a specific set of properties and capabilities, focused on creating unique lightweight solutions. 1. Allows **a swarm of multi-threaded processes to -[ACID]((https://en.wikipedia.org/wiki/ACID))ly read and update** several +[ACID](https://en.wikipedia.org/wiki/ACID)ly read and update** several key-value [maps](https://en.wikipedia.org/wiki/Associative_array) and [multimaps](https://en.wikipedia.org/wiki/Multimap) in a locally-shared database. @@ -59,6 +74,7 @@ neglected in favour of write performance. 7. Supports Linux, Windows, MacOS, Android, iOS, FreeBSD, DragonFly, Solaris, OpenSolaris, OpenIndiana, NetBSD, OpenBSD and other systems compliant with **POSIX.1-2008**. + Historically, _libmdbx_ is a deeply revised and extended descendant of the amazing @@ -66,16 +82,19 @@ Historically, _libmdbx_ is a deeply revised and extended descendant of the amazi _libmdbx_ inherits all benefits from _LMDB_, but resolves some issues and adds [a set of improvements](#improvements-beyond-lmdb). + The next version is under active non-public development from scratch and will be -released as _**MithrilDB**_ and `libmithrildb` for libraries & packages. +released as **MithrilDB** and `libmithrildb` for libraries & packages. Admittedly mythical [Mithril](https://en.wikipedia.org/wiki/Mithril) is resembling silver but being stronger and lighter than steel. Therefore _MithrilDB_ is a rightly relevant name. - > _MithrilDB_ will be radically different from _libmdbx_ by the new - > database format and API based on C++17, as well as the [Apache 2.0 - > License](https://www.apache.org/licenses/LICENSE-2.0). The goal of this - > revolution is to provide a clearer and robust API, add more features and - > new valuable properties of the database. + + > _MithrilDB_ will be radically different from _libmdbx_ by the new + > database format and API based on C++17, as well as the [Apache 2.0 + > License](https://www.apache.org/licenses/LICENSE-2.0). The goal of this + > revolution is to provide a clearer and robust API, add more features and + > new valuable properties of the database. + ----- @@ -185,6 +204,7 @@ databases"](https://github.com/coreos/bbolt#comparison-with-other-databases) which is also (mostly) applicable to _libmdbx_. + Improvements beyond LMDB @@ -200,45 +220,52 @@ the user's point of view. ## Added Features 1. Keys could be more than 2 times longer than _LMDB_. - > For DB with default page size _libmdbx_ support keys up to 2022 bytes - > and up to 32742 bytes for 64K page size. _LMDB_ allows key size up to - > 511 bytes and may silently loses data with large values. + + > For DB with default page size _libmdbx_ support keys up to 2022 bytes + > and up to 32742 bytes for 64K page size. _LMDB_ allows key size up to + > 511 bytes and may silently loses data with large values. 2. Up to 30% faster than _LMDB_ in [CRUD](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete) benchmarks. - > Benchmarks of the in-[tmpfs](https://en.wikipedia.org/wiki/Tmpfs) scenarios, - > that tests the speed of the engine itself, showned that _libmdbx_ 10-20% faster than _LMDB_, - > and up to 30% faster when _libmdbx_ compiled with specific build options - > which downgrades several runtime checks to be match with LMDB behaviour. - > - > These and other results could be easily reproduced with [ioArena](https://github.com/pmwkaa/ioarena) just by `make bench-quartet` command, - > including comparisons with [RockDB](https://en.wikipedia.org/wiki/RocksDB) - > and [WiredTiger](https://en.wikipedia.org/wiki/WiredTiger). + + > Benchmarks of the in-[tmpfs](https://en.wikipedia.org/wiki/Tmpfs) scenarios, + > that tests the speed of the engine itself, showned that _libmdbx_ 10-20% faster than _LMDB_, + > and up to 30% faster when _libmdbx_ compiled with specific build options + > which downgrades several runtime checks to be match with LMDB behaviour. + > + > These and other results could be easily reproduced with [ioArena](https://github.com/pmwkaa/ioarena) just by `make bench-quartet` command, + > including comparisons with [RockDB](https://en.wikipedia.org/wiki/RocksDB) + > and [WiredTiger](https://en.wikipedia.org/wiki/WiredTiger). 3. Automatic on-the-fly database size adjustment, both increment and reduction. - > _libmdbx_ manages the database size according to parameters specified - > by `mdbx_env_set_geometry()` function, - > ones include the growth step and the truncation threshold. - > - > Unfortunately, on-the-fly database size adjustment doesn't work under [Wine](https://en.wikipedia.org/wiki/Wine_(software)) - > due to its internal limitations and unimplemented functions, i.e. the `MDBX_UNABLE_EXTEND_MAPSIZE` error will be returned. + + > _libmdbx_ manages the database size according to parameters specified + > by `mdbx_env_set_geometry()` function, + > ones include the growth step and the truncation threshold. + > + > Unfortunately, on-the-fly database size adjustment doesn't work under [Wine](https://en.wikipedia.org/wiki/Wine_(software)) + > due to its internal limitations and unimplemented functions, i.e. the `MDBX_UNABLE_EXTEND_MAPSIZE` error will be returned. 4. Automatic continuous zero-overhead database compactification. - > During each commit _libmdbx_ merges a freeing pages which adjacent with the unallocated area - > at the end of file, and then truncates unused space when a lot enough of. + + > During each commit _libmdbx_ merges a freeing pages which adjacent with the unallocated area + > at the end of file, and then truncates unused space when a lot enough of. 5. The same database format for 32- and 64-bit builds. - > _libmdbx_ database format depends only on the [endianness](https://en.wikipedia.org/wiki/Endianness) but not on the [bitness](https://en.wiktionary.org/wiki/bitness). + + > _libmdbx_ database format depends only on the [endianness](https://en.wikipedia.org/wiki/Endianness) but not on the [bitness](https://en.wiktionary.org/wiki/bitness). 6. LIFO policy for Garbage Collection recycling. This can significantly increase write performance due write-back disk cache up to several times in a best case scenario. - > LIFO means that for reuse will be taken the latest becomes unused pages. - > Therefore the loop of database pages circulation becomes as short as possible. - > In other words, the set of pages, that are (over)written in memory and on disk during a series of write transactions, will be as small as possible. - > Thus creates ideal conditions for the battery-backed or flash-backed disk cache efficiency. + + > LIFO means that for reuse will be taken the latest becomes unused pages. + > Therefore the loop of database pages circulation becomes as short as possible. + > In other words, the set of pages, that are (over)written in memory and on disk during a series of write transactions, will be as small as possible. + > Thus creates ideal conditions for the battery-backed or flash-backed disk cache efficiency. 7. Fast estimation of range query result volume, i.e. how many items can be found between a `KEY1` and a `KEY2`. This is a prerequisite for build and/or optimize query execution plans. - > _libmdbx_ performs a rough estimate based on common B-tree pages of the paths from root to corresponding keys. + + > _libmdbx_ performs a rough estimate based on common B-tree pages of the paths from root to corresponding keys. 8. `mdbx_chk` utility for database integrity check. Since version 0.9.1, the utility supports checking the database using any of the three meta pages and the ability to switch to it. @@ -251,12 +278,14 @@ Since version 0.9.1, the utility supports checking the database using any of the or not, that allows to avoid copy-out before updates. 12. Extended information of whole-database, sub-databases, transactions, readers enumeration. - > _libmdbx_ provides a lot of information, including dirty and leftover pages - > for a write transaction, reading lag and holdover space for read transactions. + + > _libmdbx_ provides a lot of information, including dirty and leftover pages + > for a write transaction, reading lag and holdover space for read transactions. 13. Extended update and delete operations. - > _libmdbx_ allows one _at once_ with getting previous value - > and addressing the particular item from multi-value with the same key. + + > _libmdbx_ allows one _at once_ with getting previous value + > and addressing the particular item from multi-value with the same key. 14. Useful runtime options for tuning engine to application's requirements and use cases specific. @@ -286,9 +315,10 @@ pre-opening is not needed. 4. Returning `MDBX_EMULTIVAL` error in case of ambiguous update or delete. 5. Guarantee of database integrity even in asynchronous unordered write-to-disk mode. - > _libmdbx_ propose additional trade-off by `MDBX_SAFE_NOSYNC` with append-like manner for updates, - > that avoids database corruption after a system crash contrary to LMDB. - > Nevertheless, the `MDBX_UTTERLY_NOSYNC` mode is available to match LMDB's behaviour for `MDB_NOSYNC`. + + > _libmdbx_ propose additional trade-off by `MDBX_SAFE_NOSYNC` with append-like manner for updates, + > that avoids database corruption after a system crash contrary to LMDB. + > Nevertheless, the `MDBX_UTTERLY_NOSYNC` mode is available to match LMDB's behaviour for `MDB_NOSYNC`. 6. On **MacOS & iOS** the `fcntl(F_FULLFSYNC)` syscall is used _by default_ to synchronize data with the disk, as this is [the only way to @@ -320,7 +350,7 @@ year later _libmdbx_ was separated into a standalone project, which was [presented at Highload++ 2015 conference](http://www.highload.ru/2015/abstracts/1831.html). -Since 2017 _libmdbx_ is used in [Fast Positive Tables](https://erased_by_github.none/libfpta), +Since 2017 _libmdbx_ is used in [Fast Positive Tables](https://gitflic.ru/project/erthink/libfpta), and development is funded by [Positive Technologies](https://www.ptsecurity.com). ## Acknowledgments @@ -338,29 +368,25 @@ Usage ===== + Currently, libmdbx is only available in a [source code](https://en.wikipedia.org/wiki/Source_code) form. Packages support for common Linux distributions is planned in the future, since release the version 1.0. -## Never use tarballs nor zips automatically provided by Github ! - -Please don't use tarballs nor zips which are automatically provided by Github. -These archives do not contain version information and thus are unfit to build _libmdbx_. -Instead of ones just clone the git repository, either download a tarball or zip with the properly amalgamated source core. -Moreover, please vote for [ability of disabling auto-creation such unsuitable archives](https://github.community/t/disable-tarball). - ## Source code embedding _libmdbx_ provides two official ways for integration in source code form: 1. Using the amalgamated source code. - > The amalgamated source code includes all files required to build and - > use _libmdbx_, but not for testing _libmdbx_ itself. + + > The amalgamated source code includes all files required to build and + > use _libmdbx_, but not for testing _libmdbx_ itself. 2. Adding the complete original source code as a `git submodule`. - > This allows you to build as _libmdbx_ and testing tool. - > On the other hand, this way requires you to pull git tags, and use C++11 compiler for test tool. + + > This allows you to build as _libmdbx_ and testing tool. + > On the other hand, this way requires you to pull git tags, and use C++11 compiler for test tool. _**Please, avoid using any other techniques.**_ Otherwise, at least don't ask for support and don't name such chimeras `libmdbx`. @@ -387,8 +413,7 @@ and build options respectively. The only significant specificity is that git' tags are required to build from complete (not amalgamated) source codes. Executing **`git fetch --tags --force --prune`** is enough to get ones, -or `git fetch --unshallow --tags --prune --force` after the Github's -[`actions/checkout@v2`](https://github.com/actions/checkout) either set **`fetch-depth: 0`** for it. +and `--unshallow` or `--update-shallow` is required for shallow cloned case. So just using CMake or GNU Make in your habitual manner and feel free to fill an issue or make pull request in the case something will be @@ -558,7 +583,7 @@ from the [ios-cmake](https://github.com/leetal/ios-cmake) project. ## API description -Please refer to the online [_libmdbx_ API reference](https://erased_by_github_and_to_be_restored.todo/libmdbx/) +Please refer to the online [_libmdbx_ API reference](https://libmdbx.website.yandexcloud.net/docs) and/or see the [mdbx.h++](mdbx.h%2B%2B) and [mdbx.h](mdbx.h) headers. @@ -617,7 +642,7 @@ Here showed sum of performance metrics in 3 benchmarks: 2. Performance gap is too high to compare in any meaningful way. -![Comparison #1: Integral Performance](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-1.png) +![Comparison #1: Integral Performance](https://libmdbx.website.yandexcloud.net/img/perf-slide-1.png) -------------------------------------------------------------------------------- @@ -626,7 +651,7 @@ Here showed sum of performance metrics in 3 benchmarks: Summary performance with concurrent read/search queries in 1-2-4-8 threads on the machine with 4 logical CPUs in HyperThreading mode (i.e. actually 2 physical CPU cores). -![Comparison #2: Read Scalability](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-2.png) +![Comparison #2: Read Scalability](https://libmdbx.website.yandexcloud.net/img/perf-slide-2.png) -------------------------------------------------------------------------------- @@ -648,7 +673,7 @@ In the benchmark each transaction contains combined CRUD operations (2 inserts, 1 read, 1 update, 1 delete). Benchmark starts on an empty database and after full run the database contains 10,000 small key-value records. -![Comparison #3: Sync-write mode](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-3.png) +![Comparison #3: Sync-write mode](https://libmdbx.website.yandexcloud.net/img/perf-slide-3.png) -------------------------------------------------------------------------------- @@ -675,7 +700,7 @@ and after full run the database contains 100,000 small key-value records. -![Comparison #4: Lazy-write mode](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-4.png) +![Comparison #4: Lazy-write mode](https://libmdbx.website.yandexcloud.net/img/perf-slide-4.png) -------------------------------------------------------------------------------- @@ -699,7 +724,7 @@ In the benchmark each transaction contains combined CRUD operations (2 inserts, 1 read, 1 update, 1 delete). Benchmark starts on an empty database and after full run the database contains 10,000 small key-value records. -![Comparison #5: Async-write mode](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-5.png) +![Comparison #5: Async-write mode](https://libmdbx.website.yandexcloud.net/img/perf-slide-5.png) -------------------------------------------------------------------------------- @@ -723,6 +748,6 @@ All benchmark data is gathered by [getrusage()](http://man7.org/linux/man-pages/man2/getrusage.2.html) syscall and by scanning the data directory. -![Comparison #6: Cost comparison](https://erased_by_github_and_to_be_restored.todo/libmdbx/img/perf-slide-6.png) +![Comparison #6: Cost comparison](https://libmdbx.website.yandexcloud.net/img/perf-slide-6.png) diff --git a/docs/_toc.md b/docs/_toc.md index 97c7afd8..30734e07 100644 --- a/docs/_toc.md +++ b/docs/_toc.md @@ -1,3 +1,8 @@ +The origin repository of the project has been migrated +to [GitFlic](https://gitflic.ru/project/erthink/libmdbx) since on April 15, 2022, +the Github administration, without any warning not explanation, deleted +libmdbx along with a lot of other projects, simultaneously blocking +access to many developers. For the same reason ~~Github~~ is blacklisted forever. _The Future will (be) [Positive](https://www.ptsecurity.com). Всё будет хорошо._ @@ -23,7 +28,7 @@ each of which is divided into several sections. - The \ref mdbx.h++ header file reference Please do not hesitate to point out errors in the documentation, -including creating [PR](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/proposing-changes-to-your-work-with-pull-requests) with corrections and improvements. +including creating [merge-request](https://gitflic.ru/project/erthink/libmdbx/merge-request) with corrections and improvements. ---