The planned frontward release with new superior features on the day of 20 anniversary of [Positive Technologies](https://ptsecurty.com).
New:
----
- The `Big Foot` feature which significantly reduces GC overhead for processing large lists of retired pages from huge transactions.
Now _libmdbx_ avoid creating large chunks of PNLs (page number lists) which required a long sequences of free pages, aka large/overflow pages.
Thus avoiding searching, allocating and storing such sequences inside GC.
- Improved hot/online validation and checking of database pages both for more robustness and performance.
- New solid and fast method to latch meta-pages called `Troika`.
The minimum of memory barriers, reads, comparisons and conditional transitions are used.
- New `MDBX_VALIDATION` environment options to extra validation of DB structure and pages content for carefully/safe handling damaged or untrusted DB.
- Accelerated ×16/×8/×4 by AVX512/AVX2/SSE2/Neon implementations of search page sequences.
- Added the `gcrtime_seconds16dot16` counter to the "Page Operation Statistics" that accumulates time spent for GC searching and reclaiming.
- Copy-with-compactification now clears/zeroes unused gaps inside database pages.
- The `C` and `C++` APIs has been extended and/or refined to simplify using `wchar_t` pathnames.
On Windows the `mdbx_env_openW()`, `mdbx_env_get_pathW()`, `mdbx_env_copyW()`, `mdbx_env_open_for_recoveryW()` are available for now,
but the `mdbx_env_get_path()` has been replaced in favor of `mdbx_env_get_pathW()`.
- Added explicit error message for Buildroot's Microblaze toolchain maintainers.
- Added `MDBX_MANAGE_BUILD_FLAGS` build options for CMake.
- Speed-up internal `bsearch`/`lower_bound` implementation using branchless tactic, including workaround for CLANG x86 optimiser bug.
- A lot internal refinement and micro-optimisations.
- Internally counted volume of dirty pages (unused for now but for coming features).
Fixes:
------
- Never use modern `__cxa_thread_atexit()` on Apple's OSes.
- Don't check owner for finished transactions.
- Fixed typo in `MDBX_EINVAL` which breaks MingGW builds with CLANG.
37 files changed, 7604 insertions(+), 7417 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
The stable bugfix release.
It is planned that this will be the last release of the v0.11 branch.
New:
----
- The C++ API has been refined to simplify support for `wchar_t` in path names.
- Added explicit error message for Buildroot's Microblaze toolchain maintainers.
Fixes:
------
- Never use modern `__cxa_thread_atexit()` on Apple's OSes.
- Use `MultiByteToWideChar(CP_THREAD_ACP)` instead of `mbstowcs()`.
- Don't check owner for finished transactions.
- Fixed typo in `MDBX_EINVAL` which breaks MingGW builds with CLANG.
Minors:
-------
- Fixed variable name typo.
- Using `ldd` to check used dso.
- Added `MDBX_WEAK_IMPORT_ATTRIBUTE` macro.
- Use current transaction geometry for untouched parameters when `env_set_geometry()` called within a write transaction.
- Minor clarified `iov_page()` failure case.
14 files changed, 263 insertions(+), 252 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
The stable bugfix release.
It is planned that this will be the last release of the v0.11 branch.
Acknowledgements:
-----------------
- [Alex Sharov](https://github.com/AskAlexSharov) and Erigon team for reporting and testing.
- [Andrew Ashikhmin](https://gitflic.ru/user/yperbasis) for contributing.
New:
----
- Ability to customise `MDBX_LOCK_SUFFIX`, `MDBX_DATANAME`, `MDBX_LOCKNAME` just by predefine ones during build.
- Added to [`mdbx::env_managed`](https://libmdbx.dqdkfa.ru/group__cxx__api.html#classmdbx_1_1env__managed)'s methods a few overloads with `const char* pathname` parameter (C++ API).
Fixes:
------
- Fixed hang copy-with-compactification of a corrupted DB
or in case the volume of output pages is a multiple of `MDBX_ENVCOPY_WRITEBUF`.
- Fixed standalone non-CMake build on MacOS (`#include AvailabilityMacros.h>`).
- Fixed unexpected `MDBX_PAGE_FULL` error in rare cases with large database page sizes.
Minors:
-------
- Minor fixes Doxygen references, comments, descriptions, etc.
- Fixed copy&paste typo inside `meta_checktxnid()`.
- Minor fix `meta_checktxnid()` to avoid assertion in debug mode.
- Minor fix `mdbx_env_set_geometry()` to avoid returning `EINVAL` in particular rare cases.
- Minor refine/fix batch-get testcase for large page size.
- Added `--pagesize NN` option to long-stotastic test script.
- Updated Valgrind-suppressions file for modern GCC.
- Fixed `has no symbols` warning from Apple's ranlib.
18 files changed, 318 insertions(+), 178 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
The stable release with an important fixes and workaround for the critical macOS thread-local-storage issue.
Acknowledgements:
-----------------
- [Masatoshi Fukunaga](https://github.com/mah0x211) for [Lua bindings](https://github.com/mah0x211/lua-libmdbx).
New:
----
- Added most of transactions flags to the public API.
- Added `MDBX_NOSUCCESS_EMPTY_COMMIT` build option to return non-success result (`MDBX_RESULT_TRUE`) on empty commit.
- Reworked validation and import of DBI-handles into a transaction.
Assumes these changes will be invisible to most users, but will cause fewer surprises in complex DBI cases.
- Added ability to open DB in without-LCK (exclusive read-only) mode in case no permissions to create/write LCK-file.
Fixes:
------
- A series of fixes and improvements for automatically generated documentation (Doxygen).
- Fixed copy&paste bug with could lead to `SIGSEGV` (nullptr dereference) in the exclusive/no-lck mode.
- Fixed minor warnings from modern Apple's CLANG 13.
- Fixed minor warnings from CLANG 14 and in-development CLANG 15.
- Fixed `SIGSEGV` regression in without-LCK (exclusive read-only) mode.
- Fixed `mdbx_check_fs_local()` for CDROM case on Windows.
- Fixed nasty typo of typename which caused false `MDBX_CORRUPTED` error in a rare execution path,
when the size of the thread-ID type not equal to 8.
- Fixed write-after-free memory corruption on latest `macOS` during finalization/cleanup of thread(s) that executed read transaction(s).
> The issue was suddenly discovered by a [CI](https://en.wikipedia.org/wiki/Continuous_integration)
> after adding an iteration with macOS 11 "Big Sur", and then reproduced on recent release of macOS 12 "Monterey".
> The issue was never noticed nor reported on macOS 10 "Catalina" nor others.
> Analysis shown that the problem caused by a change in the behavior of the system library (internals of dyld and pthread)
> during thread finalization/cleanup: now a memory allocated for a `__thread` variable(s) is released
> before execution of the registered Thread-Local-Storage destructor(s),
> thus a TLS-destructor will write-after-free just by legitime dereference any `__thread` variable.
> This is unexpected crazy-like behavior since the order of resources releasing/destroying
> is not the reverse of ones acquiring/construction order. Nonetheless such surprise
> is now workarounded by using atomic compare-and-swap operations on a 64-bit signatures/cookies.
- Fixed Elbrus/E2K LCC 1.26 compiler warnings (memory model for atomic operations, etc).
Minors:
-------
- Refined `release-assets` GNU Make target.
- Added logging to `mdbx_fetch_sdb()` to help debugging complex DBI-handels use cases.
- Added explicit error message from probe of no-support for `std::filesystem`.
- Added contributors "score" table by `git fame` to generated docs.
- Added `mdbx_assert_fail()` to public API (mostly for backtracing).
- Now C++20 concepts used/enabled only when `__cpp_lib_concepts >= 202002`.
- Don't provide nor report package information if used as a CMake subproject.
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
The stable risen release after the Github's intentional malicious disaster.
We have migrated to a reliable trusted infrastructure
-----------------------------------------------------
The origin for now is at [GitFlic](https://gitflic.ru/project/erthink/libmdbx)
since on 2022-04-15 the Github administration, without any warning nor
explanation, deleted _libmdbx_ along with a lot of other projects,
simultaneously blocking access for many developers.
For the same reason ~~Github~~ is blacklisted forever.
GitFlic already support Russian and English languages, plan to support more,
including 和 中文. You are welcome!
New:
----
- Added the `tools-static` make target to build statically linked MDBX tools.
- Support for Microsoft Visual Studio 2022.
- Support build by MinGW' make from command line without CMake.
- Added `mdbx::filesystem` C++ API namespace that corresponds to `std::filesystem` or `std::experimental::filesystem`.
- Created [website](https://libmdbx.website.yandexcloud.net/) for online auto-generated documentation.
- Used `todo4recovery://erased_by_github/` for dead (or temporarily lost) resources deleted by ~~Github~~.
- Added `--loglevel=` command-line option to the `mdbx_test` tool.
- Added few fast smoke-like tests into CMake builds.
Fixes:
------
- Fixed a race between starting a transaction and creating a DBI descriptor that could lead to `SIGSEGV` in the cursor tracking code.
- Clarified description of `MDBX_EPERM` error returned from `mdbx_env_set_geometry()`.
- Fixed non-promoting the parent transaction to be dirty in case the undo of the geometry update failed during abortion of a nested transaction.
- Resolved linking issues with `libstdc++fs`/`libc++fs`/`libc++experimental` for C++ `std::filesystem` or `std::experimental::filesystem` for legacy compilers.
- Added workaround for GNU Make 3.81 and earlier.
- Added workaround for Elbrus/LCC 1.25 compiler bug of class inline `static constexpr` member field.
- [Fixed](https://github.com/ledgerwatch/erigon/issues/3874) minor assertion regression (only debug builds were affected).
- Fixed detection of `C++20` concepts accessibility.
- Fixed detection of Clang's LTO availability for Android.
- Fixed build for ARM/ARM64 by MSVC.
- Fixed non-x86 Windows builds with `MDBX_WITHOUT_MSVC_CRT=ON` and `MDBX_BUILD_SHARED_LIBRARY=ON`.
Minors:
-------
- Resolve minor MSVC warnings: avoid `/INCREMENTAL[:YES]` with `/LTCG`, `/W4` with `/W3`, the `C5105` warning.
- Switched to using `MDBX_EPERM` instead of `MDBX_RESULT_TRUE' to indicate that the geometry cannot be updated.
- Added `NULL` checking during memory allocation inside `mdbx_chk`.
- Resolved all warnings from MinGW while used without CMake.
- Added inheretable `target_include_directories()` to `CMakeLists.txt` for easy integration.
- Added build-time checks and paranoid runtime assertions for the `off_t` arguments of `fcntl()` which are used for locking.
- Added `-Wno-lto-type-mismatch` to avoid false-positive warnings from old GCC during LTO-enabled builds.
- Added checking for TID (system thread id) to avoid hang on 32-bit Bionic/Android within `pthread_mutex_lock()`.
- Reworked `MDBX_BUILD_TARGET` of CMake builds.
- Added `CMAKE_HOST_ARCH` and `CMAKE_HOST_CAN_RUN_EXECUTABLES_BUILT_FOR_TARGET`.
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
The stable release with the hotfix/workaround for a flaw of Linux 4.19 (at least) unified page/buffer cache.
See [issue#269](https://github.com/erthink/libmdbx/issues/269) for more information.
Acknowledgements:
-----------------
- [Simon Leier](https://github.com/leisim) for reporting and testing.
- [Kai Wetlesen](https://github.com/kaiwetlesen) for [RPMs](http://copr.fedorainfracloud.org/coprs/kwetlesen/libmdbx/).
- [Tullio Canepa](https://github.com/canepat) for reporting C++ API issue and contributing.
Fixes:
------
- [Added workaround](https://github.com/erthink/libmdbx/issues/269) for a flaw of Linux 4.19 (at least) unified page/buffer cache.
- [Fixed/Reworked](https://github.com/erthink/libmdbx/pull/270) move-assignment operators for "managed" classes of C++ API.
- Fixed potential `SIGSEGV` while open DB with overrided non-default page size.
- [Made](https://github.com/erthink/libmdbx/issues/267) `mdbx_env_open()` idempotence in failure cases.
- Refined/Fixed pages reservation inside `mdbx_update_gc()` to avoid non-reclamation in a rare cases.
- Fixed typo in a retained space calculation for the hsr-callback.
Minors:
-------
- Reworked functions for meta-pages, split-off non-volatile.
- Disentangled C11-atomic fences/barriers and pure-functions (with `__attribute__((__pure__))`) to avoid compiler misoptimization.
- Fixed hypotetic unaligned access to 64-bit dwords on ARM with `__ARM_FEATURE_UNALIGNED` defined.
- Reasonable paranoia that makes clarity for code readers.
- Minor fixes Doxygen references, comments, descriptions, etc.
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
The stable release with fixes for large and huge databases sized of 4..128 TiB.
Acknowledgements:
-----------------
- Ledgerwatch, Binance and Positive Technologies teams for reporting, assistance in investigation and testing.
- Alex Sharov for reporting, testing and provide resources for remote debugging/investigation.
- Kris Zyp for Deno support.
New features, extensions and improvements:
------------------------------------------
- Added treating the `UINT64_MAX` value as maximum for given option inside `mdbx_env_set_option()`.
- Added `to_hex/to_base58/to_base64::output(std::ostream&)` overloads without using temporary string objects as buffers.
- Added `--geometry-jitter=YES|no` option to the test framework.
- Added support for [Deno](https://deno.land/) support by [Kris Zyp](https://github.com/kriszyp).
Fixes:
------
- Fixed handling `MDBX_opt_rp_augment_limit` for GC's records from huge transactions (Erigon/Akula/Ethereum).
- [Fixed](https://github.com/erthink/libmdbx/issues/258) build on Android (avoid including `sys/sem.h`).
- [Fixed](https://github.com/erthink/libmdbx/pull/261) missing copy assignment operator for `mdbx::move_result`.
- Fixed missing `&` for `std::ostream &operator<<()` overloads.
- Fixed unexpected `EXDEV` (Cross-device link) error from `mdbx_env_copy()`.
- Fixed base64 encoding/decoding bugs in auxillary C++ API.
- Fixed overflow of `pgno_t` during checking PNL on 64-bit platforms.
- [Fixed](https://github.com/erthink/libmdbx/issues/260) excessive PNL checking after sort for spilling.
- Reworked checking `MAX_PAGENO` and DB upper-size geometry limit.
- [Fixed](https://github.com/erthink/libmdbx/issues/265) build for some combinations of versions of MSVC and Windows SDK.
Minors:
-------
- Added workaround for CLANG bug [D79919/PR42445](https://reviews.llvm.org/D79919).
- Fixed build test on Android (using `pthread_barrier_t` stub).
- Disabled C++20 concepts for CLANG < 14 on Android.
- Fixed minor `unused parameter` warning.
- Added CI for Android.
- Refine/cleanup internal logging.
- Refined line splitting inside hex/base58/base64 encoding to avoid `\n` at the end.
- Added workaround for modern libstdc++ with CLANG < 4.x
- Relaxed txn-check rules for auxiliary functions.
- Clarified a comments and descriptions, etc.
- Using the `-fno-semantic interposition` option to reduce the overhead to calling self own public functions.
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
Acknowledgements:
-----------------
- [gcxfd <i@rmw.link>](https://github.com/gcxfd) for reporting, contributing and testing.
- [장세연 (Чан Се Ен)](https://github.com/sasgas) for reporting and testing.
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting, testing and provide resources for remote debugging/investigation.
New features, extensions and improvements:
------------------------------------------
- [Added](https://github.com/erthink/libmdbx/issues/236) `mdbx_cursor_get_batch()`.
- [Added](https://github.com/erthink/libmdbx/issues/250) `MDBX_SET_UPPERBOUND`.
- C++ API is finalized now.
- The GC update stage has been [significantly speeded](https://github.com/erthink/libmdbx/issues/254) when fixing huge Erigon's transactions (Ethereum ecosystem).
Fixes:
------
- Disabled C++20 concepts for stupid AppleClang 13.x
- Fixed internal collision of `MDBX_SHRINK_ALLOWED` with `MDBX_ACCEDE`.
Minors:
-------
- Fixed returning `MDBX_RESULT_TRUE` (unexpected -1) from `mdbx_env_set_option()`.
- Added `mdbx_env_get_syncbytes()` and `mdbx_env_get_syncperiod()`.
- [Clarified](https://github.com/erthink/libmdbx/pull/249) description of `MDBX_INTEGERKEY`.
- Reworked/simplified `mdbx_env_sync_internal()`.
- [Fixed](https://github.com/erthink/libmdbx/issues/248) extra assertion inside `mdbx_cursor_put()` for `MDBX_DUPFIXED` cases.
- Avoiding extra looping inside `mdbx_env_info_ex()`.
- Explicitly enabled core dumps from stochastic tests scripts on Linux.
- [Fixed](https://github.com/erthink/libmdbx/issues/253) `mdbx_override_meta()` to avoid false-positive assertions.
- For compatibility reverted returning `MDBX_ENODATA`for some cases.
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Backward compatibility break:
-----------------------------
The database format signature has been changed to prevent
forward-interoperability with an previous releases, which may lead to a
[false positive diagnosis of database corruption](https://github.com/erthink/libmdbx/issues/238)
due to flaws of an old library versions.
This change is mostly invisible:
- previously versions are unable to read/write a new DBs;
- but the new release is able to handle an old DBs and will silently upgrade ones.
Acknowledgements:
-----------------
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Acknowledgements:
-----------------
- [Noel Kuntze](https://github.com/Thermi) for immediately bug reporting.
Fixes:
------
- Fixed unaligned access regression after the `#pragma pack` fix for modern compilers.
- Added UBSAN-test to CI to avoid a regression(s) similar to lately fixed.
- Fixed possibility of meta-pages clashing after manually turn to a particular meta-page using `mdbx_chk` utility.
Minors:
-------
- Refined handling of weak or invalid meta-pages while a DB opening.
- Refined providing information for the @MAIN and @GC sub-databases of a last committed modification transaction's ID.
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Acknowledgements:
-----------------
- [Artem Vorotnikov](https://github.com/vorot93) for support [Rust wrapper](https://github.com/vorot93/mdbx-rs).
- [Andrew Ashikhmin](https://github.com/yperbasis) for contributing to C++ API.
Fixes:
------
- Fixed possibility of looping update GC during transaction commit (no public issue since the problem was discovered inside [Positive Technologies](https://www.ptsecurity.ru).
- Fixed `#pragma pack` to avoid provoking some compilers to generate code with [unaligned access](https://github.com/erthink/libmdbx/issues/235).
- Fixed `noexcept` for potentially throwing `txn::put()` of C++ API.
Minors:
-------
- Added stochastic test script for checking small transactions cases.
- Removed extra transaction commit/restart inside test framework.
- In debugging builds fixed a too small (single page) by default DB shrink threshold.
Acknowledgements:
-----------------
- [Francisco Vallarino](https://github.com/fjvallarino) for [Haskell bindings for libmdbx](https://hackage.haskell.org/package/libmdbx).
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
- [Andrea Lanfranchi](https://github.com/AndreaLanfranchi) for contributing.
Extensions and improvements:
----------------------------
- Added `cursor::erase()` overloads for `key` and for `key-value`.
- Resolve minor Coverity Scan issues (no fixes but some hint/comment were added).
- Resolve minor UndefinedBehaviorSanitizer issues (no fixes but some workaround were added).
Fixes:
------
- Always setup `madvise` while opening DB (fixes https://github.com/erthink/libmdbx/issues/231).
- Fixed checking legacy `P_DIRTY` flag (`0x10`) for nested/sub-pages.
Minors:
-------
- Fixed getting revision number from middle of history during amalgamation (GNU Makefile).
- Fixed search GCC tools for LTO (CMake scripts).
- Fixed/reorder dirs list for search CLANG tools for LTO (CMake scripts).
- Fixed/workarounds for CLANG < 9.x
- Fixed CMake warning about compatibility with 3.8.2
Acknowledgements:
-----------------
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
- [Andrea Lanfranchi](https://github.com/AndreaLanfranchi) for reporting bugs.
- [Lionel Debroux](https://github.com/debrouxl) for fuzzing tests and reporting bugs.
- [Sergey Fedotov](https://github.com/SergeyFromHell/) for [`node-mdbx` NodeJS bindings](https://www.npmjs.com/package/node-mdbx).
- [Kris Zyp](https://github.com/kriszyp) for [`lmdbx-store` NodeJS bindings](https://github.com/kriszyp/lmdbx-store).
- [Noel Kuntze](https://github.com/Thermi) for [draft Python bindings](https://github.com/erthink/libmdbx/commits/python-bindings).
New features, extensions and improvements:
------------------------------------------
- [Allow to predefine/override `MDBX_BUILD_TIMESTAMP` for builds reproducibility](https://github.com/erthink/libmdbx/issues/201).
- Added options support for `long-stochastic` script.
- Avoided `MDBX_TXN_FULL` error for large transactions when possible.
- The `MDBX_READERS_LIMIT` increased to `32767`.
- Raise `MDBX_TOO_LARGE` under Valgrind/ASAN if being opened DB is 100 larger than RAM (to avoid hangs and OOM).
- Minimized the size of poisoned/unpoisoned regions to avoid Valgrind/ASAN stuck.
- Added more workarounds for QEMU for testing builds for 32-bit platforms, Alpha and Sparc architectures.
- `mdbx_chk` now skips iteration & checking of DB' records if corresponding page-tree is corrupted (to avoid `SIGSEGV`, ASAN failures, etc).
- Added more checks for [rare/fuzzing corruption cases](https://github.com/erthink/libmdbx/issues/217).
Backward compatibility break:
-----------------------------
- Use file `VERSION.txt` for version information instead of `VERSION` to avoid collision with `#include <version>`.
- Rename `slice::from/to_FOO_bytes()` to `slice::envisage_from/to_FOO_length()'.
- Rename `MDBX_TEST_EXTRA` make's variable to `MDBX_SMOKE_EXTRA`.
- Some details of the C++ API have been changed for subsequent freezing.
Fixes:
------
- Fixed excess meta-pages checks in case `mdbx_chk` is called to check the DB for a specific meta page and thus could prevent switching to the selected meta page, even if the check passed without errors.
- Fixed [recursive use of SRW-lock on Windows cause by `MDBX_NOTLS` option](https://github.com/erthink/libmdbx/issues/203).
- Fixed [log a warning during a new DB creation](https://github.com/erthink/libmdbx/issues/205).
- Fixed [false-negative `mdbx_cursor_eof()` result](https://github.com/erthink/libmdbx/issues/207).
- Fixed [`make install` with non-GNU `install` utility (OSX, BSD)](https://github.com/erthink/libmdbx/issues/208).
- Fixed [installation by `CMake` in special cases by complete use `GNUInstallDirs`'s variables](https://github.com/erthink/libmdbx/issues/209).
- Fixed [C++ Buffer issue with `std::string` and alignment](https://github.com/erthink/libmdbx/issues/191).
- Fixed `safe64_reset()` for platforms without atomic 64-bit compare-and-swap.
- Fixed hang/shutdown on big-endian platforms without `__cxa_thread_atexit()`.
- Fixed [using bad meta-pages if DB was partially/recoverable corrupted](https://github.com/erthink/libmdbx/issues/217).
- Fixed extra `noexcept` for `buffer::&assign_reference()`.
- Fixed `bootid` generation on Windows for case of change system' time.
- Fixed [test framework keygen-related issue](https://github.com/erthink/libmdbx/issues/127).
Acknowledgements:
-----------------
- [Alexey Akhunov](https://github.com/AlexeyAkhunov) and [Alex Sharov](https://github.com/AskAlexSharov) for bug reporting and testing.
New features:
-------------
- Added `-p` option to `mdbx_stat` utility for printing page operations statistic.
- Added explicit checking for and warning about using unfit github's archives.
Fixes:
------
- Fixed minor "foo not used" warnings from modern C++ compilers when building the C++ part of the library.
- Fixed confusing/messy errors when build library from unfit github's archives (https://github.com/erthink/libmdbx/issues/197).
- Fixed `#elsif` typo.
- Fixed rare unexpected `MDBX_PROBLEM` error during altering data in huge transactions due to wrong spilling/oust of dirty pages (https://github.com/erthink/libmdbx/issues/195).
Using float point (exponential quantized) representation for internal 16-bit values
of grow step and shrink threshold when huge ones
.
To minimize the impact on compatibility, only the odd values inside the upper half
of the range (i.e. 32769..65533) are used for the new representation.
Resolve https://github.com/erthink/libmdbx/issues/166
Change-Id: I273127c1842deef0d7d8885b55a805b1463556eb
Acknowledgements:
-----------------
- [Mahlon E. Smith](http://www.martini.nu/) for [FreeBSD port of libmdbx](https://svnweb.freebsd.org/ports/head/databases/mdbx/).
- [장세연](http://www.castis.com) for bug fixing and PR.
- [Clément Renault](https://github.com/Kerollmops/heed) for [Heed](https://github.com/Kerollmops/heed) fully typed Rust wrapper.
- [Alex Sharov](https://github.com/AskAlexSharov) for bug reporting.
- [Noel Kuntze](https://github.com/Thermi) for bug reporting.
Removed options and features:
-----------------------------
- Drop `MDBX_HUGE_TRANSACTIONS` build-option (now no longer required).
New features:
-------------
- Package for FreeBSD is available now by Mahlon E. Smith.
- New API functions to get/set various options (https://github.com/erthink/libmdbx/issues/128):
- the maximum number of named databases for the environment;
- the maximum number of threads/reader slots;
- threshold (since the last unsteady commit) to force flush the data buffers to disk;
- relative period (since the last unsteady commit) to force flush the data buffers to disk;
- limit to grow a list of reclaimed/recycled page's numbers for finding a sequence of contiguous pages for large data items;
- limit to grow a cache of dirty pages for reuse in the current transaction;
- limit of a pre-allocated memory items for dirty pages;
- limit of dirty pages for a write transaction;
- initial allocation size for dirty pages list of a write transaction;
- maximal part of the dirty pages may be spilled when necessary;
- minimal part of the dirty pages should be spilled when necessary;
- how much of the parent transaction dirty pages will be spilled while start each child transaction;
- Unlimited/Dynamic size of retired and dirty page lists (https://github.com/erthink/libmdbx/issues/123).
- Added `-p` option (purge subDB before loading) to `mdbx_load` tool.
- Reworked spilling of large transaction and committing of nested transactions:
- page spilling code reworked to avoid the flaws and bugs inherited from LMDB;
- limit for number of dirty pages now is controllable at runtime;
- a spilled pages, including overflow/large pages, now can be reused and refunded/compactified in nested transactions;
- more effective refunding/compactification especially for the loosed page cache.
- Added `MDBX_ENABLE_REFUND` and `MDBX_PNL_ASCENDING` internal/advanced build options.
- Added `mdbx_default_pagesize()` function.
- Better support architectures with a weak/relaxed memory consistency model (ARM, AARCH64, PPC, MIPS, RISC-V, etc) by means [C11 atomics](https://en.cppreference.com/w/c/atomic).
- Speed up page number lists and dirty page lists (https://github.com/erthink/libmdbx/issues/132).
- Added `LIBMDBX_NO_EXPORTS_LEGACY_API` build option.
Fixes:
------
- Fixed missing cleanup (null assigned) in the C++ commit/abort (https://github.com/erthink/libmdbx/pull/143).
- Fixed `mdbx_realloc()` for case of nullptr and `MDBX_AVOID_CRT=ON` for Windows.
- Fixed the possibility to use invalid and renewed (closed & re-opened, dropped & re-created) DBI-handles (https://github.com/erthink/libmdbx/issues/146).
- Fixed 4-byte aligned access to 64-bit integers, including access to the `bootid` meta-page's field (https://github.com/erthink/libmdbx/issues/153).
- Fixed minor/potential memory leak during page flushing and unspilling.
- Fixed handling states of cursors's and subDBs's for nested transactions.
- Fixed page leak in extra rare case the list of retired pages changed during update GC on transaction commit.
- Fixed assertions to avoid false-positive UB detection by CLANG/LLVM (https://github.com/erthink/libmdbx/issues/153).
- Fixed `MDBX_TXN_FULL` and regressive `MDBX_KEYEXIST` during large transaction commit with `MDBX_LIFORECLAIM` (https://github.com/erthink/libmdbx/issues/123).
- Fixed auto-recovery (`weak->steady` with the same boot-id) when Database size at last weak checkpoint is large than at last steady checkpoint.
- Fixed operation on systems with unusual small/large page size, including PowerPC (https://github.com/erthink/libmdbx/issues/157).
TODO:
-----
- Engage new terminology (https://github.com/erthink/libmdbx/issues/137).
- Resolve few TODOs (https://github.com/erthink/libmdbx/issues/124, https://github.com/erthink/libmdbx/issues/127, https://github.com/erthink/libmdbx/issues/115).
- Finalize C++ API.
- Packages for [ROSA Linux](https://www.rosalinux.ru/), [ALT Linux](https://www.altlinux.org/), Fedora/RHEL, Debian/Ubuntu.
Change-Id: I414b8ef2e4b90e04fb344779c0e3f1b4bd1c06be
This done better support architectures with a weak/relaxed memory consistency model (ARM, AARCH64, PPC, MIPS, RISC-V, etc).
Change-Id: Iee831c8dc564f1d027ff84b0d6daa559325d5a9b
Historically, the page header provides 4-byte data alignment.
Therefore, unfortunately, the meta page data is also aligned on a 4-byte boundary, but contains 64-bit values.
This commit eliminates potentially unsafe access (SPARC, MIPS, etc) to these 64-bit values aligned on a 4-byte boundary.
Thus, a build with the `-fsanitize=undefined` now passes the tests both with CLANG 11 and GCC 10.
Change-Id: Ie441103e53ed96fd40507d8c0be0689e3fee69f5
Acknowledgements:
-----------------
- Jens Alfke (Mobile Architect at [Couchbase](https://www.couchbase.com/)) for [NimDBX](https://github.com/snej/nimdbx).
- Clément Renault (CTO at [MeiliSearch](https://www.meilisearch.com/)) for [mdbx-rs](https://github.com/Kerollmops/mdbx-rs).
- Alex Sharov (Go-Lang Teach Lead at [TurboGeth/Ethereum](https://ethereum.org/)) for an extreme test cases and bug reporting.
- George Hazan (CTO at [Miranda NG](https://www.miranda-ng.org/)) for bug reporting.
- [Positive Technologies](https://www.ptsecurity.com/) for funding and [The Standoff](https://standoff365.com/).
Added features:
---------------
- Provided package for [buildroot](https://buildroot.org/).
- Binding for Nim is [available](https://github.com/snej/nimdbx) now by Jens Alfke.
- Added `mdbx_env_delete()` for deletion an environment files in a proper and multiprocess-safe way.
- Added `mdbx_txn_commit_ex()` with collecting latency information.
- Fast completion pure nested transactions.
- Added `LIBMDBX_INLINE_API` macro and inline versions of some API functions.
- Added `mdbx_cursor_copy()` function.
- Extended tests for checking cursor tracking.
- Added `MDBX_SET_LOWERBOUND` operation for `mdbx_cursor_get()`.
Fixes:
------
- Fixed missing installation of `mdbx.h++`.
- Fixed use of obsolete `__noreturn`.
- Fixed use of `yield` instruction on ARM if unsupported.
- Added pthread workaround for buggy toolchain/cmake/buildroot.
- Fixed use of `pthread_yield()` for non-GLIBC.
- Fixed use of `RegGetValueA()` on Windows 2000/XP.
- Fixed use of `GetTickCount64()` on Windows 2000/XP.
- Fixed opening DB on a network shares (in the exclusive mode).
- Fixed copy&paste typos.
- Fixed minor false-positive GCC warning.
- Added workaround for broken `DEFINE_ENUM_FLAG_OPERATORS` from Windows SDK.
- Fixed cursor state after multimap/dupsort repeated deletes (https://github.com/erthink/libmdbx/issues/121).
- Added `SIGPIPE` suppression for internal thread during `mdbx_env_copy()`.
- Fixed extra-rare `MDBX_KEY_EXIST` error during `mdbx_commit()` (https://github.com/erthink/libmdbx/issues/131).
- Fixed spilled pages checking (https://github.com/erthink/libmdbx/issues/126).
- Fixed `mdbx_load` for 'plain text' and without `-s name` cases (https://github.com/erthink/libmdbx/issues/136).
- Fixed save/restore/commit of cursors for nested transactions.
- Fixed cursors state in rare/special cases (move next beyond end-of-data, after deletion and so on).
- Added workaround for MSVC 19.28 (Visual Studio 16.8) (but may still hang during compilation).
- Fixed paranoidal Clang C++ UB for bitwise operations with flags defined by enums.
- Fixed large pages checking (for compatibility and to avoid false-positive errors from `mdbx_chk`).
- Added workaround for Wine (https://github.com/miranda-ng/miranda-ng/issues/1209).
- Fixed `ERROR_NOT_SUPPORTED` while opening DB by UNC pathnames (https://github.com/miranda-ng/miranda-ng/issues/2627).
- Added handling `EXCEPTION_POSSIBLE_DEADLOCK` condition for Windows.
TODO for v0.9.3:
----------------
- Engage new terminology (https://github.com/erthink/libmdbx/issues/137).
- Rework/speedup the implementation of the dirty page list (lazy compactification, lazy sorting via merge).
- Resolve few TODOs (https://github.com/erthink/libmdbx/issues/123, https://github.com/erthink/libmdbx/issues/124, https://github.com/erthink/libmdbx/issues/127, https://github.com/erthink/libmdbx/issues/128, https://github.com/erthink/libmdbx/issues/132, https://github.com/erthink/libmdbx/issues/115).
- Finalize C++ API (few typos and trivia bugs are still likely for now).
- Packages for ROSA Linux, ALT Linux, Fedora/RHEL, Debian/Ubuntu.
Change-Id: I1f0ac4505052f169e36b326243e5d56497b9c037
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Added features:
- Preliminary C++ API with support for C++17 polymorphic allocators.
- [Online C++ API reference](https://erthink.github.io/libmdbx/) by Doxygen.
- Quick reference for Insert/Update/Delete operations.
- Explicit `MDBX_SYNC_DURABLE` to sync modes for API clarity.
- Explicit `MDBX_ALLDUPS` and `MDBX_UPSERT` for API clarity.
- Support for read transactions preparation (`MDBX_TXN_RDONLY_PREPARE` flag).
- Support for cursor preparation/(pre)allocation and reusing (`mdbx_cursor_create()` and `mdbx_cursor_bind()` functions).
- Support for checking database using specified meta-page (see `mdbx_chk -h`).
- Support for turn to the specific meta-page after checking (see `mdbx_chk -h`).
- Support for explicit reader threads (de)registration.
- The `mdbx_txn_break()` function to explicitly mark a transaction as broken.
- Improved handling of corrupted databases by `mdbx_chk` utility and `mdbx_walk_tree()` function.
- Improved DB corruption detection by checking parent-page-txnid.
- Improved opening large DB (> 4Gb) from 32-bit code.
- Provided `pure-function` and `const-function` attributes to C API.
- Support for user-settable context for transactions & cursors.
- Revised API and documentation related to Handle-Slow-Readers callback feature.
Deprecated functions and flags:
- For clarity and API simplification the `MDBX_MAPASYNC` flag is deprecated.
Just use `MDBX_SAFE_NOSYNC` or `MDBX_UTTERLY_NOSYNC` instead of it.
- `MDBX_oom_func`, `mdbx_env_set_oomfunc()` and `mdbx_env_get_oomfunc()`
replaced with `MDBX_hsr_func`, `mdbx_env_get_hsr` and `mdbx_env_get_hsr()`.
Fixes:
- Fix `mdbx_strerror()` for `MDBX_BUSY` error (no error description is returned).
- Fix update internal meta-geo information in read-only mode (`EACCESS` or `EBADFD` error).
- Fix `mdbx_page_get()` null-defer when DB corrupted (crash by `SIGSEGV`).
- Fix `mdbx_env_open()` for re-opening after non-fatal errors (`mdbx_chk` unexpected failures).
- Workaround for MSVC 19.27 `static_assert()` bug.
- Doxygen descriptions and refinement.
- Update Valgrind's suppressions.
- Workaround to avoid infinite loop of 'nested' testcase on MIPS under QEMU.
- Fix a lot of typos & spelling (Thanks to Josh Soref for PR).
- Fix `getopt()` messages for Windows (Thanks to Andrey Sporaw for reporting).
- Fix MSVC compiler version requirements (Thanks to Andrey Sporaw for reporting).
- Workarounds for QEMU's bugs to run tests for cross-builded library under QEMU.
- Now C++ compiler optional for building by CMake.
TODO for next version(s):
- Rework/speedup the implementation of the dirty page list (lazy compactification, lazy sorting via merge).
- Finalize C++ API (few typos and trivia bugs are likely for now).
- Packages for ROSA Linux, ALT Linux, Fedora/RHEL, Debian/Ubuntu.
Change-Id: I668d77e4545d444b60708b8c64a66ce43b2c56a0
