Backward compatibility break:
-----------------------------
The database format signature has been changed to prevent
forward-interoperability with an previous releases, which may lead to a
[false positive diagnosis of database corruption](https://github.com/erthink/libmdbx/issues/238)
due to flaws of an old library versions.
This change is mostly invisible:
- previously versions are unable to read/write a new DBs;
- but the new release is able to handle an old DBs and will silently upgrade ones.
Acknowledgements:
-----------------
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Acknowledgements:
-----------------
- [Noel Kuntze](https://github.com/Thermi) for immediately bug reporting.
Fixes:
------
- Fixed unaligned access regression after the `#pragma pack` fix for modern compilers.
- Added UBSAN-test to CI to avoid a regression(s) similar to lately fixed.
- Fixed possibility of meta-pages clashing after manually turn to a particular meta-page using `mdbx_chk` utility.
Minors:
-------
- Refined handling of weak or invalid meta-pages while a DB opening.
- Refined providing information for the @MAIN and @GC sub-databases of a last committed modification transaction's ID.
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Acknowledgements:
-----------------
- [Artem Vorotnikov](https://github.com/vorot93) for support [Rust wrapper](https://github.com/vorot93/mdbx-rs).
- [Andrew Ashikhmin](https://github.com/yperbasis) for contributing to C++ API.
Fixes:
------
- Fixed possibility of looping update GC during transaction commit (no public issue since the problem was discovered inside [Positive Technologies](https://www.ptsecurity.ru).
- Fixed `#pragma pack` to avoid provoking some compilers to generate code with [unaligned access](https://github.com/erthink/libmdbx/issues/235).
- Fixed `noexcept` for potentially throwing `txn::put()` of C++ API.
Minors:
-------
- Added stochastic test script for checking small transactions cases.
- Removed extra transaction commit/restart inside test framework.
- In debugging builds fixed a too small (single page) by default DB shrink threshold.
Acknowledgements:
-----------------
- [Francisco Vallarino](https://github.com/fjvallarino) for [Haskell bindings for libmdbx](https://hackage.haskell.org/package/libmdbx).
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
- [Andrea Lanfranchi](https://github.com/AndreaLanfranchi) for contributing.
Extensions and improvements:
----------------------------
- Added `cursor::erase()` overloads for `key` and for `key-value`.
- Resolve minor Coverity Scan issues (no fixes but some hint/comment were added).
- Resolve minor UndefinedBehaviorSanitizer issues (no fixes but some workaround were added).
Fixes:
------
- Always setup `madvise` while opening DB (fixes https://github.com/erthink/libmdbx/issues/231).
- Fixed checking legacy `P_DIRTY` flag (`0x10`) for nested/sub-pages.
Minors:
-------
- Fixed getting revision number from middle of history during amalgamation (GNU Makefile).
- Fixed search GCC tools for LTO (CMake scripts).
- Fixed/reorder dirs list for search CLANG tools for LTO (CMake scripts).
- Fixed/workarounds for CLANG < 9.x
- Fixed CMake warning about compatibility with 3.8.2
Acknowledgements:
-----------------
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
- [Andrea Lanfranchi](https://github.com/AndreaLanfranchi) for reporting bugs.
- [Lionel Debroux](https://github.com/debrouxl) for fuzzing tests and reporting bugs.
- [Sergey Fedotov](https://github.com/SergeyFromHell/) for [`node-mdbx` NodeJS bindings](https://www.npmjs.com/package/node-mdbx).
- [Kris Zyp](https://github.com/kriszyp) for [`lmdbx-store` NodeJS bindings](https://github.com/kriszyp/lmdbx-store).
- [Noel Kuntze](https://github.com/Thermi) for [draft Python bindings](https://github.com/erthink/libmdbx/commits/python-bindings).
New features, extensions and improvements:
------------------------------------------
- [Allow to predefine/override `MDBX_BUILD_TIMESTAMP` for builds reproducibility](https://github.com/erthink/libmdbx/issues/201).
- Added options support for `long-stochastic` script.
- Avoided `MDBX_TXN_FULL` error for large transactions when possible.
- The `MDBX_READERS_LIMIT` increased to `32767`.
- Raise `MDBX_TOO_LARGE` under Valgrind/ASAN if being opened DB is 100 larger than RAM (to avoid hangs and OOM).
- Minimized the size of poisoned/unpoisoned regions to avoid Valgrind/ASAN stuck.
- Added more workarounds for QEMU for testing builds for 32-bit platforms, Alpha and Sparc architectures.
- `mdbx_chk` now skips iteration & checking of DB' records if corresponding page-tree is corrupted (to avoid `SIGSEGV`, ASAN failures, etc).
- Added more checks for [rare/fuzzing corruption cases](https://github.com/erthink/libmdbx/issues/217).
Backward compatibility break:
-----------------------------
- Use file `VERSION.txt` for version information instead of `VERSION` to avoid collision with `#include <version>`.
- Rename `slice::from/to_FOO_bytes()` to `slice::envisage_from/to_FOO_length()'.
- Rename `MDBX_TEST_EXTRA` make's variable to `MDBX_SMOKE_EXTRA`.
- Some details of the C++ API have been changed for subsequent freezing.
Fixes:
------
- Fixed excess meta-pages checks in case `mdbx_chk` is called to check the DB for a specific meta page and thus could prevent switching to the selected meta page, even if the check passed without errors.
- Fixed [recursive use of SRW-lock on Windows cause by `MDBX_NOTLS` option](https://github.com/erthink/libmdbx/issues/203).
- Fixed [log a warning during a new DB creation](https://github.com/erthink/libmdbx/issues/205).
- Fixed [false-negative `mdbx_cursor_eof()` result](https://github.com/erthink/libmdbx/issues/207).
- Fixed [`make install` with non-GNU `install` utility (OSX, BSD)](https://github.com/erthink/libmdbx/issues/208).
- Fixed [installation by `CMake` in special cases by complete use `GNUInstallDirs`'s variables](https://github.com/erthink/libmdbx/issues/209).
- Fixed [C++ Buffer issue with `std::string` and alignment](https://github.com/erthink/libmdbx/issues/191).
- Fixed `safe64_reset()` for platforms without atomic 64-bit compare-and-swap.
- Fixed hang/shutdown on big-endian platforms without `__cxa_thread_atexit()`.
- Fixed [using bad meta-pages if DB was partially/recoverable corrupted](https://github.com/erthink/libmdbx/issues/217).
- Fixed extra `noexcept` for `buffer::&assign_reference()`.
- Fixed `bootid` generation on Windows for case of change system' time.
- Fixed [test framework keygen-related issue](https://github.com/erthink/libmdbx/issues/127).
Acknowledgements:
-----------------
- [Alexey Akhunov](https://github.com/AlexeyAkhunov) and [Alex Sharov](https://github.com/AskAlexSharov) for bug reporting and testing.
New features:
-------------
- Added `-p` option to `mdbx_stat` utility for printing page operations statistic.
- Added explicit checking for and warning about using unfit github's archives.
Fixes:
------
- Fixed minor "foo not used" warnings from modern C++ compilers when building the C++ part of the library.
- Fixed confusing/messy errors when build library from unfit github's archives (https://github.com/erthink/libmdbx/issues/197).
- Fixed `#elsif` typo.
- Fixed rare unexpected `MDBX_PROBLEM` error during altering data in huge transactions due to wrong spilling/oust of dirty pages (https://github.com/erthink/libmdbx/issues/195).
Using float point (exponential quantized) representation for internal 16-bit values
of grow step and shrink threshold when huge ones
.
To minimize the impact on compatibility, only the odd values inside the upper half
of the range (i.e. 32769..65533) are used for the new representation.
Resolve https://github.com/erthink/libmdbx/issues/166
Change-Id: I273127c1842deef0d7d8885b55a805b1463556eb
Acknowledgements:
-----------------
- [Mahlon E. Smith](http://www.martini.nu/) for [FreeBSD port of libmdbx](https://svnweb.freebsd.org/ports/head/databases/mdbx/).
- [장세연](http://www.castis.com) for bug fixing and PR.
- [Clément Renault](https://github.com/Kerollmops/heed) for [Heed](https://github.com/Kerollmops/heed) fully typed Rust wrapper.
- [Alex Sharov](https://github.com/AskAlexSharov) for bug reporting.
- [Noel Kuntze](https://github.com/Thermi) for bug reporting.
Removed options and features:
-----------------------------
- Drop `MDBX_HUGE_TRANSACTIONS` build-option (now no longer required).
New features:
-------------
- Package for FreeBSD is available now by Mahlon E. Smith.
- New API functions to get/set various options (https://github.com/erthink/libmdbx/issues/128):
- the maximum number of named databases for the environment;
- the maximum number of threads/reader slots;
- threshold (since the last unsteady commit) to force flush the data buffers to disk;
- relative period (since the last unsteady commit) to force flush the data buffers to disk;
- limit to grow a list of reclaimed/recycled page's numbers for finding a sequence of contiguous pages for large data items;
- limit to grow a cache of dirty pages for reuse in the current transaction;
- limit of a pre-allocated memory items for dirty pages;
- limit of dirty pages for a write transaction;
- initial allocation size for dirty pages list of a write transaction;
- maximal part of the dirty pages may be spilled when necessary;
- minimal part of the dirty pages should be spilled when necessary;
- how much of the parent transaction dirty pages will be spilled while start each child transaction;
- Unlimited/Dynamic size of retired and dirty page lists (https://github.com/erthink/libmdbx/issues/123).
- Added `-p` option (purge subDB before loading) to `mdbx_load` tool.
- Reworked spilling of large transaction and committing of nested transactions:
- page spilling code reworked to avoid the flaws and bugs inherited from LMDB;
- limit for number of dirty pages now is controllable at runtime;
- a spilled pages, including overflow/large pages, now can be reused and refunded/compactified in nested transactions;
- more effective refunding/compactification especially for the loosed page cache.
- Added `MDBX_ENABLE_REFUND` and `MDBX_PNL_ASCENDING` internal/advanced build options.
- Added `mdbx_default_pagesize()` function.
- Better support architectures with a weak/relaxed memory consistency model (ARM, AARCH64, PPC, MIPS, RISC-V, etc) by means [C11 atomics](https://en.cppreference.com/w/c/atomic).
- Speed up page number lists and dirty page lists (https://github.com/erthink/libmdbx/issues/132).
- Added `LIBMDBX_NO_EXPORTS_LEGACY_API` build option.
Fixes:
------
- Fixed missing cleanup (null assigned) in the C++ commit/abort (https://github.com/erthink/libmdbx/pull/143).
- Fixed `mdbx_realloc()` for case of nullptr and `MDBX_AVOID_CRT=ON` for Windows.
- Fixed the possibility to use invalid and renewed (closed & re-opened, dropped & re-created) DBI-handles (https://github.com/erthink/libmdbx/issues/146).
- Fixed 4-byte aligned access to 64-bit integers, including access to the `bootid` meta-page's field (https://github.com/erthink/libmdbx/issues/153).
- Fixed minor/potential memory leak during page flushing and unspilling.
- Fixed handling states of cursors's and subDBs's for nested transactions.
- Fixed page leak in extra rare case the list of retired pages changed during update GC on transaction commit.
- Fixed assertions to avoid false-positive UB detection by CLANG/LLVM (https://github.com/erthink/libmdbx/issues/153).
- Fixed `MDBX_TXN_FULL` and regressive `MDBX_KEYEXIST` during large transaction commit with `MDBX_LIFORECLAIM` (https://github.com/erthink/libmdbx/issues/123).
- Fixed auto-recovery (`weak->steady` with the same boot-id) when Database size at last weak checkpoint is large than at last steady checkpoint.
- Fixed operation on systems with unusual small/large page size, including PowerPC (https://github.com/erthink/libmdbx/issues/157).
TODO:
-----
- Engage new terminology (https://github.com/erthink/libmdbx/issues/137).
- Resolve few TODOs (https://github.com/erthink/libmdbx/issues/124, https://github.com/erthink/libmdbx/issues/127, https://github.com/erthink/libmdbx/issues/115).
- Finalize C++ API.
- Packages for [ROSA Linux](https://www.rosalinux.ru/), [ALT Linux](https://www.altlinux.org/), Fedora/RHEL, Debian/Ubuntu.
Change-Id: I414b8ef2e4b90e04fb344779c0e3f1b4bd1c06be
This done better support architectures with a weak/relaxed memory consistency model (ARM, AARCH64, PPC, MIPS, RISC-V, etc).
Change-Id: Iee831c8dc564f1d027ff84b0d6daa559325d5a9b
Historically, the page header provides 4-byte data alignment.
Therefore, unfortunately, the meta page data is also aligned on a 4-byte boundary, but contains 64-bit values.
This commit eliminates potentially unsafe access (SPARC, MIPS, etc) to these 64-bit values aligned on a 4-byte boundary.
Thus, a build with the `-fsanitize=undefined` now passes the tests both with CLANG 11 and GCC 10.
Change-Id: Ie441103e53ed96fd40507d8c0be0689e3fee69f5
Acknowledgements:
-----------------
- Jens Alfke (Mobile Architect at [Couchbase](https://www.couchbase.com/)) for [NimDBX](https://github.com/snej/nimdbx).
- Clément Renault (CTO at [MeiliSearch](https://www.meilisearch.com/)) for [mdbx-rs](https://github.com/Kerollmops/mdbx-rs).
- Alex Sharov (Go-Lang Teach Lead at [TurboGeth/Ethereum](https://ethereum.org/)) for an extreme test cases and bug reporting.
- George Hazan (CTO at [Miranda NG](https://www.miranda-ng.org/)) for bug reporting.
- [Positive Technologies](https://www.ptsecurity.com/) for funding and [The Standoff](https://standoff365.com/).
Added features:
---------------
- Provided package for [buildroot](https://buildroot.org/).
- Binding for Nim is [available](https://github.com/snej/nimdbx) now by Jens Alfke.
- Added `mdbx_env_delete()` for deletion an environment files in a proper and multiprocess-safe way.
- Added `mdbx_txn_commit_ex()` with collecting latency information.
- Fast completion pure nested transactions.
- Added `LIBMDBX_INLINE_API` macro and inline versions of some API functions.
- Added `mdbx_cursor_copy()` function.
- Extended tests for checking cursor tracking.
- Added `MDBX_SET_LOWERBOUND` operation for `mdbx_cursor_get()`.
Fixes:
------
- Fixed missing installation of `mdbx.h++`.
- Fixed use of obsolete `__noreturn`.
- Fixed use of `yield` instruction on ARM if unsupported.
- Added pthread workaround for buggy toolchain/cmake/buildroot.
- Fixed use of `pthread_yield()` for non-GLIBC.
- Fixed use of `RegGetValueA()` on Windows 2000/XP.
- Fixed use of `GetTickCount64()` on Windows 2000/XP.
- Fixed opening DB on a network shares (in the exclusive mode).
- Fixed copy&paste typos.
- Fixed minor false-positive GCC warning.
- Added workaround for broken `DEFINE_ENUM_FLAG_OPERATORS` from Windows SDK.
- Fixed cursor state after multimap/dupsort repeated deletes (https://github.com/erthink/libmdbx/issues/121).
- Added `SIGPIPE` suppression for internal thread during `mdbx_env_copy()`.
- Fixed extra-rare `MDBX_KEY_EXIST` error during `mdbx_commit()` (https://github.com/erthink/libmdbx/issues/131).
- Fixed spilled pages checking (https://github.com/erthink/libmdbx/issues/126).
- Fixed `mdbx_load` for 'plain text' and without `-s name` cases (https://github.com/erthink/libmdbx/issues/136).
- Fixed save/restore/commit of cursors for nested transactions.
- Fixed cursors state in rare/special cases (move next beyond end-of-data, after deletion and so on).
- Added workaround for MSVC 19.28 (Visual Studio 16.8) (but may still hang during compilation).
- Fixed paranoidal Clang C++ UB for bitwise operations with flags defined by enums.
- Fixed large pages checking (for compatibility and to avoid false-positive errors from `mdbx_chk`).
- Added workaround for Wine (https://github.com/miranda-ng/miranda-ng/issues/1209).
- Fixed `ERROR_NOT_SUPPORTED` while opening DB by UNC pathnames (https://github.com/miranda-ng/miranda-ng/issues/2627).
- Added handling `EXCEPTION_POSSIBLE_DEADLOCK` condition for Windows.
TODO for v0.9.3:
----------------
- Engage new terminology (https://github.com/erthink/libmdbx/issues/137).
- Rework/speedup the implementation of the dirty page list (lazy compactification, lazy sorting via merge).
- Resolve few TODOs (https://github.com/erthink/libmdbx/issues/123, https://github.com/erthink/libmdbx/issues/124, https://github.com/erthink/libmdbx/issues/127, https://github.com/erthink/libmdbx/issues/128, https://github.com/erthink/libmdbx/issues/132, https://github.com/erthink/libmdbx/issues/115).
- Finalize C++ API (few typos and trivia bugs are still likely for now).
- Packages for ROSA Linux, ALT Linux, Fedora/RHEL, Debian/Ubuntu.
Change-Id: I1f0ac4505052f169e36b326243e5d56497b9c037
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Added features:
- Preliminary C++ API with support for C++17 polymorphic allocators.
- [Online C++ API reference](https://erthink.github.io/libmdbx/) by Doxygen.
- Quick reference for Insert/Update/Delete operations.
- Explicit `MDBX_SYNC_DURABLE` to sync modes for API clarity.
- Explicit `MDBX_ALLDUPS` and `MDBX_UPSERT` for API clarity.
- Support for read transactions preparation (`MDBX_TXN_RDONLY_PREPARE` flag).
- Support for cursor preparation/(pre)allocation and reusing (`mdbx_cursor_create()` and `mdbx_cursor_bind()` functions).
- Support for checking database using specified meta-page (see `mdbx_chk -h`).
- Support for turn to the specific meta-page after checking (see `mdbx_chk -h`).
- Support for explicit reader threads (de)registration.
- The `mdbx_txn_break()` function to explicitly mark a transaction as broken.
- Improved handling of corrupted databases by `mdbx_chk` utility and `mdbx_walk_tree()` function.
- Improved DB corruption detection by checking parent-page-txnid.
- Improved opening large DB (> 4Gb) from 32-bit code.
- Provided `pure-function` and `const-function` attributes to C API.
- Support for user-settable context for transactions & cursors.
- Revised API and documentation related to Handle-Slow-Readers callback feature.
Deprecated functions and flags:
- For clarity and API simplification the `MDBX_MAPASYNC` flag is deprecated.
Just use `MDBX_SAFE_NOSYNC` or `MDBX_UTTERLY_NOSYNC` instead of it.
- `MDBX_oom_func`, `mdbx_env_set_oomfunc()` and `mdbx_env_get_oomfunc()`
replaced with `MDBX_hsr_func`, `mdbx_env_get_hsr` and `mdbx_env_get_hsr()`.
Fixes:
- Fix `mdbx_strerror()` for `MDBX_BUSY` error (no error description is returned).
- Fix update internal meta-geo information in read-only mode (`EACCESS` or `EBADFD` error).
- Fix `mdbx_page_get()` null-defer when DB corrupted (crash by `SIGSEGV`).
- Fix `mdbx_env_open()` for re-opening after non-fatal errors (`mdbx_chk` unexpected failures).
- Workaround for MSVC 19.27 `static_assert()` bug.
- Doxygen descriptions and refinement.
- Update Valgrind's suppressions.
- Workaround to avoid infinite loop of 'nested' testcase on MIPS under QEMU.
- Fix a lot of typos & spelling (Thanks to Josh Soref for PR).
- Fix `getopt()` messages for Windows (Thanks to Andrey Sporaw for reporting).
- Fix MSVC compiler version requirements (Thanks to Andrey Sporaw for reporting).
- Workarounds for QEMU's bugs to run tests for cross-builded library under QEMU.
- Now C++ compiler optional for building by CMake.
TODO for next version(s):
- Rework/speedup the implementation of the dirty page list (lazy compactification, lazy sorting via merge).
- Finalize C++ API (few typos and trivia bugs are likely for now).
- Packages for ROSA Linux, ALT Linux, Fedora/RHEL, Debian/Ubuntu.
Change-Id: I668d77e4545d444b60708b8c64a66ce43b2c56a0
