1767 Commits

Author SHA1 Message Date
Леонид Юрьев (Leonid Yuriev)
9d8fc7b984 mdbx: minor mix result caching inside the get_reasonable_db_maxsize().
Change-Id: I9763ebb323d4f66e702e07f708e269e2479880ac
2022-03-05 14:31:02 +03:00
Леонид Юрьев (Leonid Yuriev)
44fb240955 mdbx: merge branch issue-269 into the devel branch. 2022-03-04 17:46:43 +03:00
Леонид Юрьев (Leonid Yuriev)
bfea3ca9fb mdbx: minor refine mdbx_page_search() internals. 2022-03-04 17:38:25 +03:00
Леонид Юрьев (Leonid Yuriev)
7ade182d64 mdbx: minor simplify/refine mdbx_env_set_geometry() internals. 2022-03-04 17:38:25 +03:00
Леонид Юрьев (Leonid Yuriev)
4adc7aa58d mdbx: add few checks to mdbx_validate_meta(). 2022-03-04 11:53:14 +03:00
Леонид Юрьев (Leonid Yuriev)
110cf09cf8 mdbx: simplify core-dbs update inside mdbx_txn_commit_ex(). 2022-03-04 11:49:54 +03:00
Леонид Юрьев (Leonid Yuriev)
fb25648b9c mdbx: force mdbx_page_get() to inline. 2022-03-04 11:49:01 +03:00
Леонид Юрьев (Leonid Yuriev)
78170a5750 mdbx: minor reduce/lowering debug "update oldest" logging. 2022-03-04 11:41:17 +03:00
Леонид Юрьев (Leonid Yuriev)
71d07b3a8e mdbx: adding cursors tracking/lists for read-only transactions.
Briefly, this commit fixes a missed flaw:
 - Cursor tracking is required to replacing shaded pages and adjusting the positions in writing transactions;
 - Thus, historically, an internal linked list was maintained for a read-write transactions, but not for a read-only.
   For this reason, the API for using cursors should be different for writing and reading transactions;
 - However, the libmdbx's API has been significantly improved, including the ability to reuse cursors and a uniform cursors behavior for any kind of transactions.
   My mistake is that due to working with MithrilDB, I forgot to make a same changes to libmdbx.

Fixes https://github.com/erthink/libmdbx/issues/272.
2022-03-03 15:00:28 +03:00
Леонид Юрьев (Leonid Yuriev)
96c93ac2f1 mdbx++: refine mdbx::env::operate_parameters() and related. 2022-03-03 14:13:10 +03:00
Леонид Юрьев (Leonid Yuriev)
6d61b18325 mdbx: fix auxiliary mdbx_txn_valgrind().
Fixed https://github.com/erthink/libmdbx/issues/273.
2022-02-27 15:42:48 +03:00
Леонид Юрьев (Leonid Yuriev)
d01e44db0c
mdbx: release v0.11.5
The stable release with the hotfix/workaround for a flaw of Linux 4.19 (at least) unified page/buffer cache.
See [issue#269](https://github.com/erthink/libmdbx/issues/269) for more information.

Acknowledgements:
-----------------

 - [Simon Leier](https://github.com/leisim) for reporting and testing.
 - [Kai Wetlesen](https://github.com/kaiwetlesen) for [RPMs](http://copr.fedorainfracloud.org/coprs/kwetlesen/libmdbx/).
 - [Tullio Canepa](https://github.com/canepat) for reporting C++ API issue and contributing.

Fixes:
------

 - [Added workaround](https://github.com/erthink/libmdbx/issues/269) for a flaw of Linux 4.19 (at least) unified page/buffer cache.
 - [Fixed/Reworked](https://github.com/erthink/libmdbx/pull/270) move-assignment operators for "managed" classes of C++ API.
 - Fixed potential `SIGSEGV` while open DB with overrided non-default page size.
 - [Made](https://github.com/erthink/libmdbx/issues/267) `mdbx_env_open()` idempotence in failure cases.
 - Refined/Fixed pages reservation inside `mdbx_update_gc()` to avoid non-reclamation in a rare cases.
 - Fixed typo in a retained space calculation for the hsr-callback.

Minors:
-------

 - Reworked functions for meta-pages, split-off non-volatile.
 - Disentangled C11-atomic fences/barriers and pure-functions (with `__attribute__((__pure__))`) to avoid compiler misoptimization.
 - Fixed hypotetic unaligned access to 64-bit dwords on ARM with `__ARM_FEATURE_UNALIGNED` defined.
 - Reasonable paranoia that makes clarity for code readers.
 - Minor fixes Doxygen references, comments, descriptions, etc.

Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
2022-02-23 20:06:30 +03:00
Леонид Юрьев (Leonid Yuriev)
6b45498985 mdbx: temporary workaround/hotfix for a flaw of Linux 4.19 (at least) unified page/buffer cache.
Temporary workaround for https://github.com/erthink/libmdbx/issues/269.
2022-02-22 23:58:57 +03:00
Леонид Юрьев (Leonid Yuriev)
77f236db2a mdbx: clarify loglevel descriptions. 2022-02-17 02:30:16 +03:00
Леонид Юрьев (Leonid Yuriev)
a0728023cd mdbx: fix typo in a retained space calculation for the hsr-callback. 2022-02-16 15:44:06 +03:00
Леонид Юрьев (Leonid Yuriev)
3705d705d3 mdbx: refine/fix pages reserving inside mdbx_updage_gc() to avoid non-reclamation in a rare cases. 2022-02-16 12:18:53 +03:00
Леонид Юрьев (Leonid Yuriev)
72bc655ece mdbx++: fix env::is_empty() to check leaf-pages instead of branch ones.
Seems like an auto-completion mistake.
Thanks to https://igor@t.me/libmdbx for reporing.
2022-02-15 18:11:31 +03:00
Леонид Юрьев (Leonid Yuriev)
c5f1f73fca mdbx: avoid extra looping inside mdbx_page_alloc() for mdbx_prep_backlog(). 2022-02-15 03:17:05 +03:00
Леонид Юрьев (Leonid Yuriev)
925a673d57 mdbx: make mdbx_env_open() idempotence in failure cases.
Fixes https://github.com/erthink/libmdbx/issues/267.
2022-02-13 20:30:57 +03:00
Леонид Юрьев (Leonid Yuriev)
c27787eb31 mdbx: fix potential SIGSEGV for DB with non-default page size.
Related to https://github.com/erthink/libmdbx/issues/267.
2022-02-13 17:04:10 +03:00
Леонид Юрьев (Leonid Yuriev)
2b6fd968d2 mdbx: rework functions for meta-pages, split-off non-volatile, more const. 2022-02-09 22:56:29 +03:00
Леонид Юрьев (Leonid Yuriev)
ef7b4289c0 mdbx: rework unaligned access.
The three points:
 - disentangle C11-atomic fences/barriers and pure-functions (with `__attribute__((__pure__))`) to avoid compiler misoptimization;
 - fix hypotetic unaligned access to 64-bit dwords on ARM with `__ARM_FEATURE_UNALIGNED` defined;
 - reasonable paranoia that makes clarity for code readers.
2022-02-09 22:56:28 +03:00
Леонид Юрьев (Leonid Yuriev)
daa7f04f61
mdbx: release v0.11.4
The stable release with fixes for large and huge databases sized of 4..128 TiB.

Acknowledgements:
-----------------

 - Ledgerwatch, Binance and Positive Technologies teams for reporting, assistance in investigation and testing.
 - Alex Sharov for reporting, testing and provide resources for remote debugging/investigation.
 - Kris Zyp for Deno support.

New features, extensions and improvements:
------------------------------------------

 - Added treating the `UINT64_MAX` value as maximum for given option inside `mdbx_env_set_option()`.
 - Added `to_hex/to_base58/to_base64::output(std::ostream&)` overloads without using temporary string objects as buffers.
 - Added `--geometry-jitter=YES|no` option to the test framework.
 - Added support for [Deno](https://deno.land/) support by [Kris Zyp](https://github.com/kriszyp).

Fixes:
------

 - Fixed handling `MDBX_opt_rp_augment_limit` for GC's records from huge transactions (Erigon/Akula/Ethereum).
 - [Fixed](https://github.com/erthink/libmdbx/issues/258) build on Android (avoid including `sys/sem.h`).
 - [Fixed](https://github.com/erthink/libmdbx/pull/261) missing copy assignment operator for `mdbx::move_result`.
 - Fixed missing `&` for `std::ostream &operator<<()` overloads.
 - Fixed unexpected `EXDEV` (Cross-device link) error from `mdbx_env_copy()`.
 - Fixed base64 encoding/decoding bugs in auxillary C++ API.
 - Fixed overflow of `pgno_t` during checking PNL on 64-bit platforms.
 - [Fixed](https://github.com/erthink/libmdbx/issues/260) excessive PNL checking after sort for spilling.
 - Reworked checking `MAX_PAGENO` and DB upper-size geometry limit.
 - [Fixed](https://github.com/erthink/libmdbx/issues/265) build for some combinations of versions of  MSVC and Windows SDK.

Minors:
-------

 - Added workaround for CLANG bug [D79919/PR42445](https://reviews.llvm.org/D79919).
 - Fixed build test on Android (using `pthread_barrier_t` stub).
 - Disabled C++20 concepts for CLANG < 14 on Android.
 - Fixed minor `unused parameter` warning.
 - Added CI for Android.
 - Refine/cleanup internal logging.
 - Refined line splitting inside hex/base58/base64 encoding to avoid `\n` at the end.
 - Added workaround for modern libstdc++ with CLANG < 4.x
 - Relaxed txn-check rules for auxiliary functions.
 - Clarified a comments and descriptions, etc.
 - Using the `-fno-semantic interposition` option to reduce the overhead to calling self own public functions.

Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
2022-02-02 20:56:28 +03:00
Leonid Yuriev
5aeb7ccf25 mdbx: fix extra casting inside mdbx_limits_dbsize_max(). 2022-02-02 14:09:03 +03:00
Leonid Yuriev
79e1cc3bbc mdbx: add workaround for old MSVC and/or old Windows SDK.
Resolves https://github.com/erthink/libmdbx/issues/265.

Change-Id: I295b5d9d5ecd670ccf258791bf87379a3ca17f21
2022-02-01 13:28:59 +03:00
Leonid Yuriev
bd51e181fb mdbx: refine comment for mdbx_update_gc(). 2022-01-31 21:36:57 +03:00
Леонид Юрьев (Leonid Yuriev)
d93a13294a mdbx: explicit disable -Wattributes by #pragma for GCC < 9. 2022-01-27 17:20:12 +03:00
Leonid Yuriev
fe5199b9d1 mdbx: fix/rework PNL checking during search, refine/simplify spilled pages search.
Fixes https://github.com/erthink/libmdbx/issues/260.

Change-Id: I8824f6af2f770c46668ee577aeeeb93b30b17a13
2022-01-26 13:21:21 +03:00
Leonid Yuriev
9ae56332d7 mdbx: fix Coverity warnings. 2022-01-22 18:01:31 +03:00
Leonid Yuriev
03d828834b mdbx: schedule the release of v0.11.4 for 2022-01-28. 2022-01-21 15:43:17 +03:00
Leonid Yuriev
cd2c5f594c mdbx: rework checking MAX_PAGENO and DB upper-size limit.
Related to https://github.com/erthink/libmdbx/issues/260.
2022-01-21 02:25:27 +03:00
Leonid Yuriev
ddef217047 mdbx: relax txn-check rules for auxiliary functions. 2022-01-21 02:13:48 +03:00
Leonid Yuriev
3e28cc2a25 mdbx: allow mdbx_dbi_flags_ex() for errored transactions. 2022-01-21 02:13:48 +03:00
Leonid Yuriev
8779f665dc mdbx: fix excessive PNL checking after sort for spilling.
This bug triggered only in the DEBUG builds or when the assertion checking is forcibly enabled.
It does not affect any core logic and cannot lead to DB corruption, data loss, and so on.

Fixes https://github.com/erthink/libmdbx/issues/260.
2022-01-20 15:25:35 +03:00
Leonid Yuriev
52552ebfe3 mdbx++: fix MSVC warnings/errors. 2022-01-19 21:08:01 +03:00
Leonid Yuriev
a6acc9d1a3 mdbx++: fix minor comment typo.
Change-Id: Ic169523218b366be0688ab92b76c4120df829ba3
2022-01-17 11:06:27 +03:00
Leonid Yuriev
c588af6aca mdbx: update Copyright year.
Change-Id: Ib3e4c0ac94882c4b2a1e167dd98e6e26dbdf48a4
2022-01-17 11:06:14 +03:00
Leonid Yuriev
f2995ac75b mdbx++: add assertions for destination-overflow to from/to hex/base58/base64::write_bytes().
Change-Id: I5a1f23dcb1dc1942f9d72966a78080eacfacf546
2022-01-15 18:52:12 +03:00
Leonid Yuriev
b5b0a9a284 mdbx++: add to_hex/to_base58/to_base64::output(std::ostream&) without using temporary objects/buffers/strings.
Change-Id: Ideffd0e7f450307e14d780dcdeb2458c1c7e4c18
2022-01-15 18:52:12 +03:00
Leonid Yuriev
b139d8165b mdbx++: refine line splitting inside to_hex/to_base58/to_base64 to avoid \n at the end.
Change-Id: I53fb1f31ac5fc41567826a9035b0080fd62c8ade
2022-01-15 18:51:56 +03:00
Leonid Yuriev
ef2e390b10 mdbx++: fix from_base64::write_bytes().
Change-Id: I6c90997bffab45019a77edcf2a9dd35ae80069d3
2022-01-15 18:51:56 +03:00
Leonid Yuriev
51e6d4645c mdbx++: fix to_base64::write_bytes() for tail bytes.
Change-Id: I60defa11b7e44dcf2ac840160ab0c833484c9d39
2022-01-15 17:22:18 +03:00
Leonid Yuriev
2151be6cde mdbx: merge branch 'issue-260' into devel. 2022-01-15 15:31:45 +03:00
Leonid Yuriev
7a06cac680 mdbx: fix unexpected EXDEV (Cross-device link) error from mdbx_env_copy()
Fallback to `write()` if `copy_file_range()` syscall returns `EXDEV`.
2022-01-14 12:39:38 +03:00
Leonid Yuriev
1813bf9e53 mdbx: avoid 32-bit pgno_t overflow during checking PNL on 64-bit platforms.
Related to https://github.com/erthink/libmdbx/issues/260.
2022-01-13 15:42:00 +03:00
Leonid Yuriev
406cafb642 mdbx: treat the UINT64_MAX value as maximum for given option inside mdbx_env_set_option(). 2022-01-06 00:37:08 +03:00
Leonid Yuriev
b19ebf0c2e mdbx: fix minor unused parameter warning. 2022-01-05 17:47:45 +03:00
Leonid Yuriev
066bb696e3 mdbx: avoid including sys/sem.h when MDBX_LOCKING != MDBX_LOCKING_SYSV.
Fixes https://github.com/erthink/libmdbx/issues/258.
2022-01-05 17:47:09 +03:00
Leonid Yuriev
024900ee9c mdbx: fix/refine handling options.rp_augment_limit detent while loading GC records. 2022-01-05 11:03:21 +03:00
Leonid Yuriev
ab83d173ac mdbx: minor refine logging for internal GC-related ops. 2022-01-05 10:50:39 +03:00