The stable risen release after the Github's intentional malicious disaster.
We have migrated to a reliable trusted infrastructure
-----------------------------------------------------
The origin for now is at [GitFlic](https://gitflic.ru/project/erthink/libmdbx)
since on 2022-04-15 the Github administration, without any warning nor
explanation, deleted _libmdbx_ along with a lot of other projects,
simultaneously blocking access for many developers.
For the same reason ~~Github~~ is blacklisted forever.
GitFlic already support Russian and English languages, plan to support more,
including 和 中文. You are welcome!
New:
----
- Added the `tools-static` make target to build statically linked MDBX tools.
- Support for Microsoft Visual Studio 2022.
- Support build by MinGW' make from command line without CMake.
- Added `mdbx::filesystem` C++ API namespace that corresponds to `std::filesystem` or `std::experimental::filesystem`.
- Created [website](https://libmdbx.website.yandexcloud.net/) for online auto-generated documentation.
- Used `todo4recovery://erased_by_github/` for dead (or temporarily lost) resources deleted by ~~Github~~.
- Added `--loglevel=` command-line option to the `mdbx_test` tool.
- Added few fast smoke-like tests into CMake builds.
Fixes:
------
- Fixed a race between starting a transaction and creating a DBI descriptor that could lead to `SIGSEGV` in the cursor tracking code.
- Clarified description of `MDBX_EPERM` error returned from `mdbx_env_set_geometry()`.
- Fixed non-promoting the parent transaction to be dirty in case the undo of the geometry update failed during abortion of a nested transaction.
- Resolved linking issues with `libstdc++fs`/`libc++fs`/`libc++experimental` for C++ `std::filesystem` or `std::experimental::filesystem` for legacy compilers.
- Added workaround for GNU Make 3.81 and earlier.
- Added workaround for Elbrus/LCC 1.25 compiler bug of class inline `static constexpr` member field.
- [Fixed](https://github.com/ledgerwatch/erigon/issues/3874) minor assertion regression (only debug builds were affected).
- Fixed detection of `C++20` concepts accessibility.
- Fixed detection of Clang's LTO availability for Android.
- Fixed build for ARM/ARM64 by MSVC.
- Fixed non-x86 Windows builds with `MDBX_WITHOUT_MSVC_CRT=ON` and `MDBX_BUILD_SHARED_LIBRARY=ON`.
Minors:
-------
- Resolve minor MSVC warnings: avoid `/INCREMENTAL[:YES]` with `/LTCG`, `/W4` with `/W3`, the `C5105` warning.
- Switched to using `MDBX_EPERM` instead of `MDBX_RESULT_TRUE' to indicate that the geometry cannot be updated.
- Added `NULL` checking during memory allocation inside `mdbx_chk`.
- Resolved all warnings from MinGW while used without CMake.
- Added inheretable `target_include_directories()` to `CMakeLists.txt` for easy integration.
- Added build-time checks and paranoid runtime assertions for the `off_t` arguments of `fcntl()` which are used for locking.
- Added `-Wno-lto-type-mismatch` to avoid false-positive warnings from old GCC during LTO-enabled builds.
- Added checking for TID (system thread id) to avoid hang on 32-bit Bionic/Android within `pthread_mutex_lock()`.
- Reworked `MDBX_BUILD_TARGET` of CMake builds.
- Added `CMAKE_HOST_ARCH` and `CMAKE_HOST_CAN_RUN_EXECUTABLES_BUILT_FOR_TARGET`.
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
The stable release with the hotfix/workaround for a flaw of Linux 4.19 (at least) unified page/buffer cache.
See [issue#269](https://github.com/erthink/libmdbx/issues/269) for more information.
Acknowledgements:
-----------------
- [Simon Leier](https://github.com/leisim) for reporting and testing.
- [Kai Wetlesen](https://github.com/kaiwetlesen) for [RPMs](http://copr.fedorainfracloud.org/coprs/kwetlesen/libmdbx/).
- [Tullio Canepa](https://github.com/canepat) for reporting C++ API issue and contributing.
Fixes:
------
- [Added workaround](https://github.com/erthink/libmdbx/issues/269) for a flaw of Linux 4.19 (at least) unified page/buffer cache.
- [Fixed/Reworked](https://github.com/erthink/libmdbx/pull/270) move-assignment operators for "managed" classes of C++ API.
- Fixed potential `SIGSEGV` while open DB with overrided non-default page size.
- [Made](https://github.com/erthink/libmdbx/issues/267) `mdbx_env_open()` idempotence in failure cases.
- Refined/Fixed pages reservation inside `mdbx_update_gc()` to avoid non-reclamation in a rare cases.
- Fixed typo in a retained space calculation for the hsr-callback.
Minors:
-------
- Reworked functions for meta-pages, split-off non-volatile.
- Disentangled C11-atomic fences/barriers and pure-functions (with `__attribute__((__pure__))`) to avoid compiler misoptimization.
- Fixed hypotetic unaligned access to 64-bit dwords on ARM with `__ARM_FEATURE_UNALIGNED` defined.
- Reasonable paranoia that makes clarity for code readers.
- Minor fixes Doxygen references, comments, descriptions, etc.
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
The stable release with fixes for large and huge databases sized of 4..128 TiB.
Acknowledgements:
-----------------
- Ledgerwatch, Binance and Positive Technologies teams for reporting, assistance in investigation and testing.
- Alex Sharov for reporting, testing and provide resources for remote debugging/investigation.
- Kris Zyp for Deno support.
New features, extensions and improvements:
------------------------------------------
- Added treating the `UINT64_MAX` value as maximum for given option inside `mdbx_env_set_option()`.
- Added `to_hex/to_base58/to_base64::output(std::ostream&)` overloads without using temporary string objects as buffers.
- Added `--geometry-jitter=YES|no` option to the test framework.
- Added support for [Deno](https://deno.land/) support by [Kris Zyp](https://github.com/kriszyp).
Fixes:
------
- Fixed handling `MDBX_opt_rp_augment_limit` for GC's records from huge transactions (Erigon/Akula/Ethereum).
- [Fixed](https://github.com/erthink/libmdbx/issues/258) build on Android (avoid including `sys/sem.h`).
- [Fixed](https://github.com/erthink/libmdbx/pull/261) missing copy assignment operator for `mdbx::move_result`.
- Fixed missing `&` for `std::ostream &operator<<()` overloads.
- Fixed unexpected `EXDEV` (Cross-device link) error from `mdbx_env_copy()`.
- Fixed base64 encoding/decoding bugs in auxillary C++ API.
- Fixed overflow of `pgno_t` during checking PNL on 64-bit platforms.
- [Fixed](https://github.com/erthink/libmdbx/issues/260) excessive PNL checking after sort for spilling.
- Reworked checking `MAX_PAGENO` and DB upper-size geometry limit.
- [Fixed](https://github.com/erthink/libmdbx/issues/265) build for some combinations of versions of MSVC and Windows SDK.
Minors:
-------
- Added workaround for CLANG bug [D79919/PR42445](https://reviews.llvm.org/D79919).
- Fixed build test on Android (using `pthread_barrier_t` stub).
- Disabled C++20 concepts for CLANG < 14 on Android.
- Fixed minor `unused parameter` warning.
- Added CI for Android.
- Refine/cleanup internal logging.
- Refined line splitting inside hex/base58/base64 encoding to avoid `\n` at the end.
- Added workaround for modern libstdc++ with CLANG < 4.x
- Relaxed txn-check rules for auxiliary functions.
- Clarified a comments and descriptions, etc.
- Using the `-fno-semantic interposition` option to reduce the overhead to calling self own public functions.
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
Acknowledgements:
-----------------
- [gcxfd <i@rmw.link>](https://github.com/gcxfd) for reporting, contributing and testing.
- [장세연 (Чан Се Ен)](https://github.com/sasgas) for reporting and testing.
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting, testing and provide resources for remote debugging/investigation.
New features, extensions and improvements:
------------------------------------------
- [Added](https://github.com/erthink/libmdbx/issues/236) `mdbx_cursor_get_batch()`.
- [Added](https://github.com/erthink/libmdbx/issues/250) `MDBX_SET_UPPERBOUND`.
- C++ API is finalized now.
- The GC update stage has been [significantly speeded](https://github.com/erthink/libmdbx/issues/254) when fixing huge Erigon's transactions (Ethereum ecosystem).
Fixes:
------
- Disabled C++20 concepts for stupid AppleClang 13.x
- Fixed internal collision of `MDBX_SHRINK_ALLOWED` with `MDBX_ACCEDE`.
Minors:
-------
- Fixed returning `MDBX_RESULT_TRUE` (unexpected -1) from `mdbx_env_set_option()`.
- Added `mdbx_env_get_syncbytes()` and `mdbx_env_get_syncperiod()`.
- [Clarified](https://github.com/erthink/libmdbx/pull/249) description of `MDBX_INTEGERKEY`.
- Reworked/simplified `mdbx_env_sync_internal()`.
- [Fixed](https://github.com/erthink/libmdbx/issues/248) extra assertion inside `mdbx_cursor_put()` for `MDBX_DUPFIXED` cases.
- Avoiding extra looping inside `mdbx_env_info_ex()`.
- Explicitly enabled core dumps from stochastic tests scripts on Linux.
- [Fixed](https://github.com/erthink/libmdbx/issues/253) `mdbx_override_meta()` to avoid false-positive assertions.
- For compatibility reverted returning `MDBX_ENODATA`for some cases.
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Backward compatibility break:
-----------------------------
The database format signature has been changed to prevent
forward-interoperability with an previous releases, which may lead to a
[false positive diagnosis of database corruption](https://github.com/erthink/libmdbx/issues/238)
due to flaws of an old library versions.
This change is mostly invisible:
- previously versions are unable to read/write a new DBs;
- but the new release is able to handle an old DBs and will silently upgrade ones.
Acknowledgements:
-----------------
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Acknowledgements:
-----------------
- [Noel Kuntze](https://github.com/Thermi) for immediately bug reporting.
Fixes:
------
- Fixed unaligned access regression after the `#pragma pack` fix for modern compilers.
- Added UBSAN-test to CI to avoid a regression(s) similar to lately fixed.
- Fixed possibility of meta-pages clashing after manually turn to a particular meta-page using `mdbx_chk` utility.
Minors:
-------
- Refined handling of weak or invalid meta-pages while a DB opening.
- Refined providing information for the @MAIN and @GC sub-databases of a last committed modification transaction's ID.
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Acknowledgements:
-----------------
- [Artem Vorotnikov](https://github.com/vorot93) for support [Rust wrapper](https://github.com/vorot93/mdbx-rs).
- [Andrew Ashikhmin](https://github.com/yperbasis) for contributing to C++ API.
Fixes:
------
- Fixed possibility of looping update GC during transaction commit (no public issue since the problem was discovered inside [Positive Technologies](https://www.ptsecurity.ru).
- Fixed `#pragma pack` to avoid provoking some compilers to generate code with [unaligned access](https://github.com/erthink/libmdbx/issues/235).
- Fixed `noexcept` for potentially throwing `txn::put()` of C++ API.
Minors:
-------
- Added stochastic test script for checking small transactions cases.
- Removed extra transaction commit/restart inside test framework.
- In debugging builds fixed a too small (single page) by default DB shrink threshold.
Acknowledgements:
-----------------
- [Francisco Vallarino](https://github.com/fjvallarino) for [Haskell bindings for libmdbx](https://hackage.haskell.org/package/libmdbx).
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
- [Andrea Lanfranchi](https://github.com/AndreaLanfranchi) for contributing.
Extensions and improvements:
----------------------------
- Added `cursor::erase()` overloads for `key` and for `key-value`.
- Resolve minor Coverity Scan issues (no fixes but some hint/comment were added).
- Resolve minor UndefinedBehaviorSanitizer issues (no fixes but some workaround were added).
Fixes:
------
- Always setup `madvise` while opening DB (fixes https://github.com/erthink/libmdbx/issues/231).
- Fixed checking legacy `P_DIRTY` flag (`0x10`) for nested/sub-pages.
Minors:
-------
- Fixed getting revision number from middle of history during amalgamation (GNU Makefile).
- Fixed search GCC tools for LTO (CMake scripts).
- Fixed/reorder dirs list for search CLANG tools for LTO (CMake scripts).
- Fixed/workarounds for CLANG < 9.x
- Fixed CMake warning about compatibility with 3.8.2
Acknowledgements:
-----------------
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
- [Andrea Lanfranchi](https://github.com/AndreaLanfranchi) for reporting bugs.
- [Lionel Debroux](https://github.com/debrouxl) for fuzzing tests and reporting bugs.
- [Sergey Fedotov](https://github.com/SergeyFromHell/) for [`node-mdbx` NodeJS bindings](https://www.npmjs.com/package/node-mdbx).
- [Kris Zyp](https://github.com/kriszyp) for [`lmdbx-store` NodeJS bindings](https://github.com/kriszyp/lmdbx-store).
- [Noel Kuntze](https://github.com/Thermi) for [draft Python bindings](https://github.com/erthink/libmdbx/commits/python-bindings).
New features, extensions and improvements:
------------------------------------------
- [Allow to predefine/override `MDBX_BUILD_TIMESTAMP` for builds reproducibility](https://github.com/erthink/libmdbx/issues/201).
- Added options support for `long-stochastic` script.
- Avoided `MDBX_TXN_FULL` error for large transactions when possible.
- The `MDBX_READERS_LIMIT` increased to `32767`.
- Raise `MDBX_TOO_LARGE` under Valgrind/ASAN if being opened DB is 100 larger than RAM (to avoid hangs and OOM).
- Minimized the size of poisoned/unpoisoned regions to avoid Valgrind/ASAN stuck.
- Added more workarounds for QEMU for testing builds for 32-bit platforms, Alpha and Sparc architectures.
- `mdbx_chk` now skips iteration & checking of DB' records if corresponding page-tree is corrupted (to avoid `SIGSEGV`, ASAN failures, etc).
- Added more checks for [rare/fuzzing corruption cases](https://github.com/erthink/libmdbx/issues/217).
Backward compatibility break:
-----------------------------
- Use file `VERSION.txt` for version information instead of `VERSION` to avoid collision with `#include <version>`.
- Rename `slice::from/to_FOO_bytes()` to `slice::envisage_from/to_FOO_length()'.
- Rename `MDBX_TEST_EXTRA` make's variable to `MDBX_SMOKE_EXTRA`.
- Some details of the C++ API have been changed for subsequent freezing.
Fixes:
------
- Fixed excess meta-pages checks in case `mdbx_chk` is called to check the DB for a specific meta page and thus could prevent switching to the selected meta page, even if the check passed without errors.
- Fixed [recursive use of SRW-lock on Windows cause by `MDBX_NOTLS` option](https://github.com/erthink/libmdbx/issues/203).
- Fixed [log a warning during a new DB creation](https://github.com/erthink/libmdbx/issues/205).
- Fixed [false-negative `mdbx_cursor_eof()` result](https://github.com/erthink/libmdbx/issues/207).
- Fixed [`make install` with non-GNU `install` utility (OSX, BSD)](https://github.com/erthink/libmdbx/issues/208).
- Fixed [installation by `CMake` in special cases by complete use `GNUInstallDirs`'s variables](https://github.com/erthink/libmdbx/issues/209).
- Fixed [C++ Buffer issue with `std::string` and alignment](https://github.com/erthink/libmdbx/issues/191).
- Fixed `safe64_reset()` for platforms without atomic 64-bit compare-and-swap.
- Fixed hang/shutdown on big-endian platforms without `__cxa_thread_atexit()`.
- Fixed [using bad meta-pages if DB was partially/recoverable corrupted](https://github.com/erthink/libmdbx/issues/217).
- Fixed extra `noexcept` for `buffer::&assign_reference()`.
- Fixed `bootid` generation on Windows for case of change system' time.
- Fixed [test framework keygen-related issue](https://github.com/erthink/libmdbx/issues/127).
Acknowledgements:
-----------------
- [Alexey Akhunov](https://github.com/AlexeyAkhunov) and [Alex Sharov](https://github.com/AskAlexSharov) for bug reporting and testing.
New features:
-------------
- Added `-p` option to `mdbx_stat` utility for printing page operations statistic.
- Added explicit checking for and warning about using unfit github's archives.
Fixes:
------
- Fixed minor "foo not used" warnings from modern C++ compilers when building the C++ part of the library.
- Fixed confusing/messy errors when build library from unfit github's archives (https://github.com/erthink/libmdbx/issues/197).
- Fixed `#elsif` typo.
- Fixed rare unexpected `MDBX_PROBLEM` error during altering data in huge transactions due to wrong spilling/oust of dirty pages (https://github.com/erthink/libmdbx/issues/195).
Using float point (exponential quantized) representation for internal 16-bit values
of grow step and shrink threshold when huge ones
.
To minimize the impact on compatibility, only the odd values inside the upper half
of the range (i.e. 32769..65533) are used for the new representation.
Resolve https://github.com/erthink/libmdbx/issues/166
Change-Id: I273127c1842deef0d7d8885b55a805b1463556eb
Acknowledgements:
-----------------
- [Mahlon E. Smith](http://www.martini.nu/) for [FreeBSD port of libmdbx](https://svnweb.freebsd.org/ports/head/databases/mdbx/).
- [장세연](http://www.castis.com) for bug fixing and PR.
- [Clément Renault](https://github.com/Kerollmops/heed) for [Heed](https://github.com/Kerollmops/heed) fully typed Rust wrapper.
- [Alex Sharov](https://github.com/AskAlexSharov) for bug reporting.
- [Noel Kuntze](https://github.com/Thermi) for bug reporting.
Removed options and features:
-----------------------------
- Drop `MDBX_HUGE_TRANSACTIONS` build-option (now no longer required).
New features:
-------------
- Package for FreeBSD is available now by Mahlon E. Smith.
- New API functions to get/set various options (https://github.com/erthink/libmdbx/issues/128):
- the maximum number of named databases for the environment;
- the maximum number of threads/reader slots;
- threshold (since the last unsteady commit) to force flush the data buffers to disk;
- relative period (since the last unsteady commit) to force flush the data buffers to disk;
- limit to grow a list of reclaimed/recycled page's numbers for finding a sequence of contiguous pages for large data items;
- limit to grow a cache of dirty pages for reuse in the current transaction;
- limit of a pre-allocated memory items for dirty pages;
- limit of dirty pages for a write transaction;
- initial allocation size for dirty pages list of a write transaction;
- maximal part of the dirty pages may be spilled when necessary;
- minimal part of the dirty pages should be spilled when necessary;
- how much of the parent transaction dirty pages will be spilled while start each child transaction;
- Unlimited/Dynamic size of retired and dirty page lists (https://github.com/erthink/libmdbx/issues/123).
- Added `-p` option (purge subDB before loading) to `mdbx_load` tool.
- Reworked spilling of large transaction and committing of nested transactions:
- page spilling code reworked to avoid the flaws and bugs inherited from LMDB;
- limit for number of dirty pages now is controllable at runtime;
- a spilled pages, including overflow/large pages, now can be reused and refunded/compactified in nested transactions;
- more effective refunding/compactification especially for the loosed page cache.
- Added `MDBX_ENABLE_REFUND` and `MDBX_PNL_ASCENDING` internal/advanced build options.
- Added `mdbx_default_pagesize()` function.
- Better support architectures with a weak/relaxed memory consistency model (ARM, AARCH64, PPC, MIPS, RISC-V, etc) by means [C11 atomics](https://en.cppreference.com/w/c/atomic).
- Speed up page number lists and dirty page lists (https://github.com/erthink/libmdbx/issues/132).
- Added `LIBMDBX_NO_EXPORTS_LEGACY_API` build option.
Fixes:
------
- Fixed missing cleanup (null assigned) in the C++ commit/abort (https://github.com/erthink/libmdbx/pull/143).
- Fixed `mdbx_realloc()` for case of nullptr and `MDBX_AVOID_CRT=ON` for Windows.
- Fixed the possibility to use invalid and renewed (closed & re-opened, dropped & re-created) DBI-handles (https://github.com/erthink/libmdbx/issues/146).
- Fixed 4-byte aligned access to 64-bit integers, including access to the `bootid` meta-page's field (https://github.com/erthink/libmdbx/issues/153).
- Fixed minor/potential memory leak during page flushing and unspilling.
- Fixed handling states of cursors's and subDBs's for nested transactions.
- Fixed page leak in extra rare case the list of retired pages changed during update GC on transaction commit.
- Fixed assertions to avoid false-positive UB detection by CLANG/LLVM (https://github.com/erthink/libmdbx/issues/153).
- Fixed `MDBX_TXN_FULL` and regressive `MDBX_KEYEXIST` during large transaction commit with `MDBX_LIFORECLAIM` (https://github.com/erthink/libmdbx/issues/123).
- Fixed auto-recovery (`weak->steady` with the same boot-id) when Database size at last weak checkpoint is large than at last steady checkpoint.
- Fixed operation on systems with unusual small/large page size, including PowerPC (https://github.com/erthink/libmdbx/issues/157).
TODO:
-----
- Engage new terminology (https://github.com/erthink/libmdbx/issues/137).
- Resolve few TODOs (https://github.com/erthink/libmdbx/issues/124, https://github.com/erthink/libmdbx/issues/127, https://github.com/erthink/libmdbx/issues/115).
- Finalize C++ API.
- Packages for [ROSA Linux](https://www.rosalinux.ru/), [ALT Linux](https://www.altlinux.org/), Fedora/RHEL, Debian/Ubuntu.
Change-Id: I414b8ef2e4b90e04fb344779c0e3f1b4bd1c06be
This done better support architectures with a weak/relaxed memory consistency model (ARM, AARCH64, PPC, MIPS, RISC-V, etc).
Change-Id: Iee831c8dc564f1d027ff84b0d6daa559325d5a9b
Historically, the page header provides 4-byte data alignment.
Therefore, unfortunately, the meta page data is also aligned on a 4-byte boundary, but contains 64-bit values.
This commit eliminates potentially unsafe access (SPARC, MIPS, etc) to these 64-bit values aligned on a 4-byte boundary.
Thus, a build with the `-fsanitize=undefined` now passes the tests both with CLANG 11 and GCC 10.
Change-Id: Ie441103e53ed96fd40507d8c0be0689e3fee69f5
