ISAR/libmdbx
3
0
Fork 0
mirror of https://github.com/isar/libmdbx.git synced 2025-01-04 17:54:13 +08:00
Code Issues Packages Projects Releases Wiki Activity
2,929 Commits 5 Branches 68 Tags 13 MiB
Commit Graph

1623 Commits

Author SHA1 Message Date
Leonid Yuriev
b59836e6d9 mdbx: minor refactoring: use cursor_set_result. 
Change-Id: Ice724c541bc4127aae0e9ea85b4fc1d7c1b68471
 2021-04-18 10:24:22 +03:00
Leonid Yuriev
3eb48340ad mdbx: minor refactoring: use node_result. 
Change-Id: I489b15c174644b2f414925f5bacbe83c091a0692
 2021-04-18 10:24:22 +03:00
Leonid Yuriev
b3aba4691b mdbx: minor refactoring: use page_result. 
Change-Id: I7749d9463832ce9b270d06f04f43e413d5ba26b7
 2021-04-18 10:24:22 +03:00
Leonid Yuriev
e4db019f47 mdbx: minor refine cursor_set_ex(). 
Change-Id: I0951a917c1e4c9e372f29c64fda0e1b267f48cdd
 2021-04-18 10:24:22 +03:00
Leonid Yuriev
dc7098b3fb mdbx: minor refine page_new(). 
Change-Id: Iafdea1c592c346c7841a39cdfc96436460b27bcb
 2021-04-18 10:24:22 +03:00
Leonid Yuriev
6d3ff10165 mdbx: simplify page_unspill(). 
Change-Id: Iaa182d1feaf2abb59015725f000a3d9e0c57de8d
 2021-04-18 10:24:22 +03:00
Leonid Yuriev
3622433cf4 mdbx: fix page_copy() for unspilling a large/overflow page. 
Change-Id: I46c6b6bf94c877c1c0142116c404692960dc5ab7
 2021-04-18 10:24:22 +03:00
Leonid Yuriev
9c9cdfdb6d mdbx: minor fix mdbx_is_dirty(). 
Change-Id: Ia539165ddd62a40e13fdfec48de754961681d40f
 2021-04-18 10:24:22 +03:00
Leonid Yuriev
a0a4bbaa7c mdbx: drop goto/label in prev commit. 
More for https://github.com/erthink/libmdbx/issues/164

Change-Id: Ic708c3ba51dd22d08e96d0bef4632a0beb19999b
 2021-04-18 10:24:11 +03:00
Leonid Yuriev
3e0fad1cf6 mdbx: rework (NO)READAHEAD handling. 
Resolves https://github.com/erthink/libmdbx/issues/164

---

NOTE: Seems there is a bug in the Mach/Darwin/OSX kernel,
because MADV_WILLNEED with offset != 0 may cause SIGBUS
on following access to the hinted region.

19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64

Change-Id: I11ebbf2bd35e3dba9d078be16cb5678aecf8329c
 2021-04-17 01:12:16 +03:00
Leonid Yuriev
958fd5b947 mdbx: use mp_txnid instead of P_DIRTY (squashed). 
Change-Id: I7e3174611b8819d8050b84d835fe976b87c77c0c
 2021-04-14 19:27:13 +03:00
Leonid Yuriev
f587a74597 mdbx: drop mx_dbistate. 
Change-Id: I61616ee1436ed99db93a1036c9bb7d4db7157c8f
 2021-04-14 00:26:14 +03:00
Leonid Yuriev
5e334fa830 mdbx: extract mdbx_touch_dbi(). 
Change-Id: I3b077cefe44f4c2db099b2050e1fd72a1aaa2687
 2021-04-14 00:25:15 +03:00
Leonid Yuriev
f9977975ae mdbx: remove extra TXN_DBI_CHANGED() check. 
Change-Id: If2014f7bfbfab98f3c0c357f2475da57efc50693
 2021-04-14 00:23:36 +03:00
Leonid Yuriev
e35c92eabb mdbx: refine mdbx_page_new(). 
Change-Id: Ief0cb313c39f7b11fb1cd611d9b0497c011a7a7a
 2021-04-14 00:19:34 +03:00
Leonid Yuriev
49296cad14 mdbx: refactor mdbx_drop_tree() & drop mdbx_retire pgno(). 
Change-Id: I37f33d7a11e958ccc1c48a6af31599deb677038d
 2021-04-10 22:20:31 +03:00
Leonid Yuriev
dfed1dbc17 mdbx: refactoring: merge page_retire() and page_loose(). 
Change-Id: I4b47c11eedad6ef1b83dbf3d34977cdd8fa79505
 2021-04-10 22:20:24 +03:00
Leonid Yuriev
20299f87cb mdbx: minor refine get_reasonable_db_maxsize(). 
Change-Id: I5d654940a325c0c3966a59a37615820587c52c78
 2021-04-10 17:48:31 +03:00
Leonid Yuriev
7f5cbf7dd8 mdbx: avoid float-point ops in prev commit. 
Change-Id: I7fadc5096f49502c01a60436840bb1a87dfe27bd
 2021-04-07 14:29:14 +03:00
Leonid Yuriev
c14e4235ee mdbx: guess a reasonable maximum DB size for the default upper limit of geometry. 
Fixes https://github.com/erthink/libmdbx/issues/183

Change-Id: Ic7b616e229d3008fda49e5a04121e22997ac53ea
 2021-04-07 05:23:06 +03:00
Leonid Yuriev
8ff44026c3 mdbx: fix minor MSVC warning. 
Change-Id: Ia7ec726b8a516d2115b727960cf7fbc874882cd4
 2021-04-07 04:52:38 +03:00
Leonid Yuriev
024ccf6826 mdbx: extract mdbx_get_sysraminfo(). 
Change-Id: I8821009572d275a3b5d2d8729f605b115a80e50a
 2021-04-07 04:45:53 +03:00
Leonid Yuriev
6ee62650af mdbx: add comment for unsure_equal(). 
Change-Id: I0629b5cd4812415c3a240047ec4202f33dddd189
 2021-03-30 01:41:34 +03:00
Leonid Yuriev
66df21ba78 mdbx: add a TODO/FIXME note (first r/w opening while r/o present). 
Change-Id: I5351628aa22313fffd6c2501a2aa16f2c5f573d5
 2021-03-30 01:38:24 +03:00
Leonid Yuriev
4f62b059ef mdbx: minor clarify TODO (configurable threshold to keep reserve space for large/overflow pages). 
Change-Id: I73a3847852466eb3e34927c5648435c8cf283c0e
 2021-03-30 01:34:46 +03:00
Leonid Yuriev
af9b7b5605 mdbx: MAJOR rework page splitting (squashed). 
Basically, this (squashed) commit introduces:
 - An "auto-appending" feature upon insertion for both ascending and
   descending key sequences. As a result, the optimality of page filling
   increases significantly (more densely, less slackness) while
   inserting ordered sequences of keys,
 - A "splitting at middle" for more balanced page tree on average.

---

1. Using left/middle/right tactics for finding the split point of a page:
   - If a key is inserted close to an edge of page,
     then the page splits at that edge;
   - Otherwise a page splits at the middle,
     which leads to a more balanced tree on average;
   - So I expect a better behavior on average,
     but actually effects should be studied further practically.

2. New code for calculating the midpoint of a page split.

3. APPEND-flags no longer affect choosing the page split point.

4. Added left-side splitting by inserting a pure page with a new entry.

Change-Id: Id7441acfc8c90636e3be6bc00a0df15714690f3c
 2021-03-29 00:49:34 +03:00
Leonid Yuriev
e0795227e4 mdbx: merge branch master into devel. 
Change-Id: Ic78177f8bc5ab3dc1826e4e00599f1a7d3087dda
 2021-03-29 00:00:43 +03:00
Leonid Yuriev
3ed99f8c20 mdbx: minor refine/fix cursor_check() internals. 
Change-Id: I89ea9a5336b88e7d0201a7f59d66a8bc53849172
 2021-03-28 23:34:41 +03:00
Leonid Yuriev
636c212235 mdbx: minor refine page_check(). 
Change-Id: I91cae256c47064422d26663078c8c5a6f619a197
 2021-03-28 23:34:41 +03:00
Leonid Yuriev
83cd4f7d58 mdbx: separete DKBUF and DKBUF_DEBUG. 
Change-Id: I8d90909ce9c5faec176a12e68f7bab6c2ea1d317
 2021-03-28 23:34:41 +03:00
Leonid Yuriev
bfc9921305 mdbx: change branch-node size to ~1/2 of the page size. 
Change-Id: I0f455ec04170d792ecc933aa501b4ec1ca9b08c0
 2021-03-28 23:34:41 +03:00
Leonid Yuriev
72f3a2fc3f mdbx: check zero-size of the first key on a branch-page. 
Change-Id: I2be86ea79f04ea8baea92a6e56ef6b22691eeda0
 2021-03-28 22:25:40 +03:00
Leonid Yuriev
3c389d17e8 mdbx: minor refine page_rebalance(). 
Change-Id: I066e7a22d62c45f9e80fb4735f55f02ca34d3ee3
 2021-03-28 22:25:30 +03:00
Leonid Yuriev
2524a0e749 mdbx: output page type from bad_page(). 
Change-Id: I11e4931de26015204c26c116e91e4a0e02f909b4
 2021-03-28 22:25:24 +03:00
Leonid Yuriev
8d3b878965 mdbx: fix minor spelling. 
Change-Id: Ibab66dc18c92b0ac4fa705e10befb34e268ee1f7
 2021-03-28 22:25:17 +03:00
Leonid Yuriev
9e5fe2c61d mdbx-tools: add to mdbx_chk output the number of keys/entries from a pages. 
Change-Id: I7c2d779fa89848287f8b5a0fa1ff2fc89c5a9b8c
 2021-03-28 22:24:56 +03:00
Leonid Yuriev
c77494e2aa mdbx: fix mdbx_env_set_option() for dp_initial & dp_limit. 
Fixes https://github.com/erthink/libmdbx/issues/179

Change-Id: I925104a45148112e0ce94a069a7468e2b4fc8209
 2021-03-24 19:27:31 +03:00
Leonid Yuriev
422d030820 mdbx: fix minor casting warning. 
Change-Id: I9615b3928bb1b88743b42be9d82a986255d1f074
 2021-03-19 03:16:47 +03:00
Leonid Yuriev
b3798a9116 mdbx: mdbx_node_search() micro-optimization. 
Change-Id: I48064ca284aa89b16e418afdc20d09c5dd6e5cba
 2021-03-19 00:17:14 +03:00
Leonid Yuriev
cb0ee2373d mdbx: add mdbx_cursor_set_ex() with non-optional exactp argument. 
Resolves https://github.com/erthink/libmdbx/pull/178

Change-Id: I24dc7d44116b5e2e02b5b75375be24c647a762f5
 2021-03-18 23:15:55 +03:00
Leonid Yuriev
d0b3c45f04 mdbx: refine check_dbi(). 
Change-Id: I6d51c27c3acdea1b93861f67a32cd3c1ee13dc0d
 2021-03-18 23:12:56 +03:00
Leonid Yuriev
6c9e8817b7 mdbx-posix: use MAP_NORESERVE. 
Change-Id: I6a46a5bfa5ed3fa5c3f26775fcba346f71106ad1
 2021-03-17 01:27:23 +03:00
Leonid Yuriev
0be7616521 mdbx-posix: minor around MAP_FIXED_NOREPLACE. 
Change-Id: I0991a5ed1e1706442675d9514932fad30de372f8
 2021-03-17 01:27:23 +03:00
Leonid Yuriev
0fd90de97e mdbx-posix: add check for _POSIX_MAPPED_FILES. 
Change-Id: Ie7ee8464346cd57d8cd96b6c5df23da8ed37cbbb
 2021-03-17 01:27:19 +03:00
Leonid Yuriev
7dfd3f18f8 mdbx: rework nodes/keys limitations. 
Change-Id: Id0515346d762d4554102775f26a0fc33f3c0f29e
 2021-03-17 01:18:28 +03:00
Leonid Yuriev
b164baa1f5 mdbx: add MDBX_ENABLE_MADVISE build option/macro. 
Change-Id: I6174ec62e4811e891663b8ae0f7918aa09baf5ab
 2021-03-17 01:18:18 +03:00
Leonid Yuriev
bd2c3d1c9c mdbx-tools: minor refine/cleanup. 
Change-Id: I94974246c1212cf98bf3d31419e8a0fc24c5decc
 2021-03-15 01:31:38 +03:00
Leonid Yuriev
3eb343020d mdbx: simplify check_txn_rw(). 
Change-Id: Id6a24068ffcdd29fe15a888985bd4a17f1de8ab4
 2021-03-14 16:33:06 +03:00
Leonid Yuriev
4b8b7d5a77 mdbx: refine mdbx_env_set_geometry() internals and 16-bit representation of grow/shrink values. 
More for https://github.com/erthink/libmdbx/issues/166.

Change-Id: I7390f954819309ee4a01faf587aee6b5152e44bc
 2021-03-14 03:42:52 +03:00
Leonid Yuriev
55620c1d13 mdbx: fix selecting/adjustment values inside mdbx_env_set_geometry() for implicit out-of-range cases. 
Fixes https://github.com/erthink/libmdbx/issues/170.

Change-Id: Ibc8754811aa3d9077447220f9fec7a5543e40cc4
 2021-03-13 19:55:19 +03:00
Leonid Yuriev
439cccf65f mdbx: minor fix assertion in the prev commit. 
One more for https://github.com/erthink/libmdbx/issues/171.

Change-Id: I49be3df948f3dbcc838f3dc307155ac404cfb331
 2021-03-13 19:40:18 +03:00
Leonid Yuriev
f7cd08ea48 mdbx: add an explicit/strict check that the environment is open. 
More for https://github.com/erthink/libmdbx/issues/171.

Change-Id: Ifbf7f8ba143d19162bd3ed1cf29c21c31b45f0d5
 2021-03-13 18:31:45 +03:00
Leonid Yuriev
e43cf081f1 mdbx: fix mdbx_env_info_ex() for a non-open environment case. 
Fix the https://github.com/erthink/libmdbx/issues/171.

Change-Id: Iad3d0186ef40dfd40fad1c3ae0dd97f9c1161c16
 2021-03-13 17:15:17 +03:00
Leonid Yuriev
a96b6f79c6 mdbx: fix/disallowing implicit subDB deletion via operations on MAIN_DBI. 
Change-Id: I3cd786e877f42cef2c0d5556033b2633f8a8ce62
 2021-03-13 17:11:35 +03:00
Leonid Yuriev
93f0f21a4c mdbx-tools: minor speedup mdbx_load by adding __hot/__inline for internal functions. 
Change-Id: Iee7d2562dbe184475f01ab9a7f9aae11549d7294
 2021-03-13 15:48:31 +03:00
Leonid Yuriev
8b5197b108 mdbx-tools: minor rearrange internal fields inside mdbx_load to avoid padding. 
Change-Id: Ieca06e0c031938b083fd12959812afa1bbc9e2de
 2021-03-13 15:47:59 +03:00
Leonid Yuriev
0603dd8305 mdbx-tools: more for quiet mode inside mdbx_load. 
Change-Id: I31839d530b94516c9819e6600fd808a3c97c609a
 2021-03-13 15:47:19 +03:00
Leonid Yuriev
f01e0efc2d mdbx-tools: add mdbx_drop tool. 
Change-Id: Ib7b32668c13fcef5951ff7250df57b3263e14d69
 2021-03-13 15:45:15 +03:00
Leonid Yuriev
fd021d793a mdbx: fix MSVC warning. 
More for https://github.com/erthink/libmdbx/issues/166

Change-Id: I47a2ec72831c06695e7ef4c52a77039e52a718e2
 2021-03-09 12:59:54 +03:00
Leonid Yuriev
cd90f831af mdbx: refine db growth step default. 
Change-Id: Ief80441cb578ae37d5d7cee4766e57eaacb4bbde
 2021-03-09 12:59:27 +03:00
Leonid Yuriev
8e51a10908 mdbx: packing the 16-bit representations of grow step and shrink threshold values. 
Using float point (exponential quantized) representation for internal 16-bit values
of grow step and shrink threshold when huge ones
.
To minimize the impact on compatibility, only the odd values inside the upper half
of the range (i.e. 32769..65533) are used for the new representation.

Resolve https://github.com/erthink/libmdbx/issues/166

Change-Id: I273127c1842deef0d7d8885b55a805b1463556eb
 2021-03-09 12:54:48 +03:00
Leonid Yuriev
ec95a50bb6 mdbx: fix not setting the dirty flag for @MAIN when dropping DBI. 
This fixes https://github.com/erthink/libmdbx/issues/168

Change-Id: Ida9f11fbcea092af17946b95bfe4ac58b822a80c
 2021-03-06 20:40:15 +03:00
Leonid Yuriev
b0d449565d mdbx: shrink dbi table immediately in the mdbx_dbi_close_locked(). 
Change-Id: Idaa5ec9791b5ffa4305e7f9f7a625fba5f956dd9
 2021-03-05 03:20:21 +03:00
Leonid Yuriev
de13d6c823 mdbx: merge branch 'master' into devel. 
Change-Id: Icec2e7548923ef7b3e33bf50fc20842938d5eec5
 2021-03-04 19:23:09 +03:00
Erk
b2a0279253
Fix compilation on build targets listed below MIPS 2021-03-04 10:33:12 +01:00
Leonid Yuriev
dec11e639a mdbx: fix mdbx_dpl_alloc(). 
Change-Id: Ia7d4b1866ca3623668edffd56ab83e845f16b2a5
 2021-03-03 16:24:32 +03:00
Leonid Yuriev
0cebc50291 mdbx: refine page-rebalance. 
Change-Id: Ia94379fa51eb5da1e1ec1a52cc1dd8b67ceb150a
 2021-03-03 16:05:52 +03:00
Leonid Yuriev
46dcd6e7ca mdbx: (re)alloc dpl-list at txn start. 
Change-Id: Ie35221666f09a189f5513e77d3ecb4056968334f
 2021-03-02 03:58:27 +03:00
Leonid Yuriev
8cabd99d24 mdbx: merge branch 'devel'. 
Change-Id: I2593f0229694d0bdd310975887e31651ead6dcc0
 2021-03-02 02:12:10 +03:00
Leonid Yuriev
c4b24b6a4d mdbx-tools: minor fix a condition for details output about leaf pages. 
Thanks to Alexander Gavrilov (https://github.com/dartraiden) for the hint.

Change-Id: I2a674b7d23310f8c776eef551353f79dfca7a3d8
 2021-03-01 23:09:18 +03:00
Leonid Yuriev
c8dccc9bc4 mdbx: limits the initial size of dpl-list to the current db-size. 
Change-Id: I5f575fc6168f50786b6f8a82ae020d323530a12e
 2021-02-28 01:02:16 +03:00
Leonid Yuriev
72d978ee48 mdbx: weakens checks during set the MDBX_opt_txn_dp_limit to avoid MDBX_EINVAL. 
Change-Id: I4852261d0c45b726c60792463ab698538fa447e5
 2021-02-28 01:02:16 +03:00
Leonid Yuriev
7fcf11013e mdbx: minor fix likely/unlikely inside mdbx_cursor_del(). 
Change-Id: I86cfc755eef7371ea96c0feb39bffd3ec5298b71
 2021-02-28 01:02:16 +03:00
Leonid Yuriev
3fd739ea2c mdbx: add MDBX_DISABLE_PAGECHECKS build option. 
Change-Id: Ia5e2727b74961e7f829f273225c5c1a6cc764b8e
 2021-02-07 02:53:54 +03:00
Leonid Yuriev
c89f30e485 mdbx: fix/rework C11 atomics usage to avoid performance regression. 
Resolve https://github.com/erthink/libmdbx/issues/160

Change-Id: Ic14ef8c9f1e4fb844952f51e81b58268d7939cfe
 2021-02-06 21:16:56 +03:00
Leonid Yuriev
7e7d526ed5 mdbx: minor fix txn_spill(). 
Change-Id: Ic624681d3a9e262b77f761796aabf0231019db72
 2021-02-06 19:34:26 +03:00
Leonid Yuriev
0e8c913c57 mdbx: minor refine node_search(). 
Change-Id: Ib9aaca3e7853b9986b6d70bc43a88e90d0a46c0c
 2021-02-06 19:33:07 +03:00
Leonid Yuriev
892402a5d8 mdbx: add missing unlikely() hints. 
Change-Id: I0baae944b9d2a994ad65fbb57591c2cc8d35f0d3
 2021-02-06 19:31:04 +03:00
Leonid Yuriev
0cc695e22e mdbx: minor simplify page_get(). 
Change-Id: I2a548ea008fad29da8382946dff3c7d9bffa1e85
 2021-02-06 19:30:05 +03:00
Leonid Yuriev
be220dc905 mdbx: release v0.9.3 
Acknowledgements:
-----------------
 - [Mahlon E. Smith](http://www.martini.nu/) for [FreeBSD port of libmdbx](https://svnweb.freebsd.org/ports/head/databases/mdbx/).
 - [장세연](http://www.castis.com) for bug fixing and PR.
 - [Clément Renault](https://github.com/Kerollmops/heed) for [Heed](https://github.com/Kerollmops/heed) fully typed Rust wrapper.
 - [Alex Sharov](https://github.com/AskAlexSharov) for bug reporting.
 - [Noel Kuntze](https://github.com/Thermi) for bug reporting.

Removed options and features:
-----------------------------
 - Drop `MDBX_HUGE_TRANSACTIONS` build-option (now no longer required).

New features:
-------------
 - Package for FreeBSD is available now by Mahlon E. Smith.
 - New API functions to get/set various options (https://github.com/erthink/libmdbx/issues/128):
    - the maximum number of named databases for the environment;
    - the maximum number of threads/reader slots;
    - threshold (since the last unsteady commit) to force flush the data buffers to disk;
    - relative period (since the last unsteady commit) to force flush the data buffers to disk;
    - limit to grow a list of reclaimed/recycled page's numbers for finding a sequence of contiguous pages for large data items;
    - limit to grow a cache of dirty pages for reuse in the current transaction;
    - limit of a pre-allocated memory items for dirty pages;
    - limit of dirty pages for a write transaction;
    - initial allocation size for dirty pages list of a write transaction;
    - maximal part of the dirty pages may be spilled when necessary;
    - minimal part of the dirty pages should be spilled when necessary;
    - how much of the parent transaction dirty pages will be spilled while start each child transaction;
 - Unlimited/Dynamic size of retired and dirty page lists (https://github.com/erthink/libmdbx/issues/123).
 - Added `-p` option (purge subDB before loading) to `mdbx_load` tool.
 - Reworked spilling of large transaction and committing of nested transactions:
    - page spilling code reworked to avoid the flaws and bugs inherited from LMDB;
    - limit for number of dirty pages now is controllable at runtime;
    - a spilled pages, including overflow/large pages, now can be reused and refunded/compactified in nested transactions;
    - more effective refunding/compactification especially for the loosed page cache.
 - Added `MDBX_ENABLE_REFUND` and `MDBX_PNL_ASCENDING` internal/advanced build options.
 - Added `mdbx_default_pagesize()` function.
 - Better support architectures with a weak/relaxed memory consistency model (ARM, AARCH64, PPC, MIPS, RISC-V, etc) by means [C11 atomics](https://en.cppreference.com/w/c/atomic).
 - Speed up page number lists and dirty page lists (https://github.com/erthink/libmdbx/issues/132).
 - Added `LIBMDBX_NO_EXPORTS_LEGACY_API` build option.

Fixes:
------
 - Fixed missing cleanup (null assigned) in the C++ commit/abort (https://github.com/erthink/libmdbx/pull/143).
 - Fixed `mdbx_realloc()` for case of nullptr and `MDBX_AVOID_CRT=ON` for Windows.
 - Fixed the possibility to use invalid and renewed (closed & re-opened, dropped & re-created) DBI-handles (https://github.com/erthink/libmdbx/issues/146).
 - Fixed 4-byte aligned access to 64-bit integers, including access to the `bootid` meta-page's field (https://github.com/erthink/libmdbx/issues/153).
 - Fixed minor/potential memory leak during page flushing and unspilling.
 - Fixed handling states of cursors's and subDBs's for nested transactions.
 - Fixed page leak in extra rare case the list of retired pages changed during update GC on transaction commit.
 - Fixed assertions to avoid false-positive UB detection by CLANG/LLVM (https://github.com/erthink/libmdbx/issues/153).
 - Fixed `MDBX_TXN_FULL` and regressive `MDBX_KEYEXIST` during large transaction commit with `MDBX_LIFORECLAIM` (https://github.com/erthink/libmdbx/issues/123).
 - Fixed auto-recovery (`weak->steady` with the same boot-id) when Database size at last weak checkpoint is large than at last steady checkpoint.
 - Fixed operation on systems with unusual small/large page size, including PowerPC (https://github.com/erthink/libmdbx/issues/157).

TODO:
-----
 - Engage new terminology (https://github.com/erthink/libmdbx/issues/137).
 - Resolve few TODOs (https://github.com/erthink/libmdbx/issues/124, https://github.com/erthink/libmdbx/issues/127,    https://github.com/erthink/libmdbx/issues/115).
 - Finalize C++ API.
 - Packages for [ROSA Linux](https://www.rosalinux.ru/), [ALT Linux](https://www.altlinux.org/), Fedora/RHEL, Debian/Ubuntu.

Change-Id: I414b8ef2e4b90e04fb344779c0e3f1b4bd1c06be
 2021-02-02 22:18:21 +03:00
Leonid Yuriev
8133d93678 mdbx: add LIBMDBX_NO_EXPORTS_LEGACY_API build option. 
Change-Id: I9d9dcffe63fce843e22132c3829adca57714d818
 2021-02-02 00:25:48 +03:00
Leonid Yuriev
38a559b93e mdbx: Crutches for C11-atomics compiler's bugs. 
Change-Id: I5a857aa56c831f00c4bc67c95d7aecabc29264da
 2021-02-01 20:55:41 +03:00
Leonid Yuriev
25c4df0d3e mdbx: radix sort for large chunks of PNL and DPL. 
More for https://github.com/erthink/libmdbx/issues/132

Change-Id: I19b253f78069d4ecd4ec360a12121c78f182fc09
 2021-02-01 02:14:13 +03:00
Leonid Yuriev
88bdf4b96f mdbx: avoid branches inside bsearch() loop. 
Related for https://github.com/erthink/libmdbx/issues/132

Change-Id: Ia843556cc7052e5081a98f56b43fd80e2d0a40c7
 2021-02-01 00:39:52 +03:00
Leonid Yuriev
28bd5d81d2 mdbx: incremental lazy merge sort for dirty-page list. 
The main change for https://github.com/erthink/libmdbx/issues/132

Change-Id: I9907f6abfcf77bd40c7a263f5adb5274d5445864
 2021-02-01 00:39:19 +03:00
Leonid Yuriev
0620ec2f2e mdbx: merge branch master into the devel. 
Change-Id: Iab3a0502577f6557e3f8fccc89df49aee372416e
 2021-01-31 23:21:16 +03:00
Leonid Yuriev
077989bfed mdbx: adds pre- and post- gaps to dirty page list to avoid some comparisons and conditional branches. 
More for https://github.com/erthink/libmdbx/issues/132

Change-Id: I6562c5ff6c559341bb7bb64222b126f06cc13427
 2021-01-31 23:20:49 +03:00
Leonid Yuriev
edb6d2d661 mdbx: add audit after nested txn start. 
Change-Id: I7601994d24e7d2a5b8e9b113f697e4a14b1d2e6a
 2021-01-31 23:20:49 +03:00
Leonid Yuriev
df180d1d36 mdbx: provide the gap/reserve for ability of merge sort dirty-page list. 
Preparation for fix https://github.com/erthink/libmdbx/issues/132

Change-Id: I4aac6927878eb0361588fbfcf9698ba6d2d5fc34
 2021-01-31 23:16:10 +03:00
Leonid Yuriev
8b6608a9ca mdbx: fix dirtyroom inheritance for nested transactions. 
Change-Id: If2036ab3b7909c51e54526a535d65cdee1e0b287
 2021-01-31 21:49:20 +03:00
Leonid Yuriev
9f0ff865e8 mdbx: use C11 atomics if available instead of legacy memory barriers. 
This done better support architectures with a weak/relaxed memory consistency model (ARM, AARCH64, PPC, MIPS, RISC-V, etc).

Change-Id: Iee831c8dc564f1d027ff84b0d6daa559325d5a9b
 2021-01-30 02:28:12 +03:00
Leonid Yuriev
f698f07ff9 mdbx: fix operation for unusual small/large system page size. 
More for https://github.com/erthink/libmdbx/issues/157

Change-Id: I4f2ed54b50653d0375538b82c48590d1037cd93b
 2021-01-29 21:19:58 +03:00
Leonid Yuriev
6cfac546a1 mdbx: fix mdbx_env_get_option(MDBX_opt_sync_bytes) to return number of bytes instead of pages. 
Change-Id: I815887e82946d9644b05549ec253a52b566c8565
 2021-01-29 19:55:40 +03:00
Leonid Yuriev
9c9f6faf38 mdbx: fix madvise_threshold for large system pages. 
Fixed https://github.com/erthink/libmdbx/issues/157

Change-Id: I0e880591d236b6ba96a38d2a3fa953695d9c4df0
 2021-01-29 04:58:32 +03:00
Leonid Yuriev
0a2f2e28b4 mdbx: fix minor signed/unsigned comparison warnings. 
Change-Id: Ice6f9759560d5eda514112e838d57e63207a7559
 2021-01-27 19:23:07 +03:00
Leonid Yuriev
62b2e31bf4 mdbx: fix auto-recovery weak->steady with the same boot-id (don't truncate DB to size from steady-meta). 
Change-Id: Ib1c7f845bdf49fb9101b01aa868f8567172e5590
 2021-01-27 16:48:03 +03:00
Leonid Yuriev
943bb552a2 mdbx: fix GC-slot cleanup inside mdbx_update_gc() in extra rare complex cases. 
More for 5f1b684719 and https://github.com/erthink/libmdbx/issues/123

Change-Id: I1b78d6be2babf9ffdd0b77e7b1ee53d1313977bb
 2021-01-26 10:28:39 +03:00
Leonid Yuriev
251eda6fb8 mdbx: HNY! 
Change-Id: I86c7028001b349e3d06b837dc8e2f7e60fb3b1a2
 2021-01-26 07:27:13 +03:00
Leonid Yuriev
4fd23c4716 mdbx: fix minor debug typo. 
Change-Id: Ieffd3cea351681c1d5ccf93b12953eb0d8abb6b9
 2021-01-25 23:26:53 +03:00
Leonid Yuriev
5f1b684719 mdbx: fix regression MDBX_KEYEXISTS during mdbx_commit_ex() and overwriting GC records. 
Fix regression related to https://github.com/erthink/libmdbx/issues/123 and https://github.com/erthink/libmdbx/issues/128.
Related to https://github.com/erthink/libmdbx/issues/131.

В lifo-режиме при фиксации транзакции, записи в GC могли быть перезаписаны (с утечкой страниц БД), либо могла возникать ошибка MDBX_KEYEXISTS, по следующему сценарию:
- В истории БД были две транзакции с огромным кол-вом retired pages, после которых в GC остались две соответствующие записи.
- В ходе очередной транзакции первая из огромных GC-записей попадает в переработку и образует огромный reclaimed list.
- При фиксации транзакции производится попытка разбить огромный reclaimed list на чанки размером в одну страницу. Для этого требуется много id для записей, которые в соответствии с lifo должны быть максимально близки к голове GC, т. е. получены путем переработки последних записей GC.
- В ходе переработки последних записей очередь доходит до второй огромной записи, при этом переработка прерывается, ибо иначе reclaimed list переполнится.
- Однако прерывание переработки внутри mdbx_update_gc() трактовалось как отсутствие записей в GC, поэтому список доступных просто добавлялись соответствующие id-шники.
- Если в списке доступных id-шников для помещения в GC были переработанные, то записи с id по всему списку удалялись - тогда вторая большая запись (и возможно предыдущие) удалялись, а содержащиеся в них номера страниц выпадали из оборота.
- Если же в списке доступных id-шников не было переработанных, то чистка не проводилась - тогда при последующая попытка помещения чанков reclaimed list в GC завершалась ошибкой MDBX_KEYEXISTS, которая и возвращалась из mdbx_commit_ex().

Change-Id: I3e5d40ef7950b7476da0513c6836fcba1de74879
 2021-01-25 23:23:08 +03:00
Leonid Yuriev
ee22ffb878 mdbx-tools: using walked pages count while output statistics if traversal enabled. 2021-01-25 21:35:59 +03:00
Leonid Yuriev
a9cb6a6a90 mdbx: minor fix ASAN controls. 2021-01-25 21:35:59 +03:00
Leonid Yuriev
95ae324580 mdbx: fix reclaimed-list allocation during nested txn start. 
Change-Id: Ia70e01ed69a001249ce14b4930452a469d04b824
 2021-01-24 00:49:46 +03:00
Leonid Yuriev
4ae2a107bf mdbx: add MDBX_opt_loose_limit runtime option. 
More for https://github.com/erthink/libmdbx/issues/128

Change-Id: I298bf1eef24575df72615ee07e93f5073405e8e0
 2021-01-22 23:52:03 +03:00
Leonid Yuriev
dc34041600 mdbx: alter defaults for MDBX_opt_rp_augment_limit and MDBX_opt_txn_dp_limit runtime options. 
Change-Id: I08cdbfb18089a3fed4b32d931ca4423027a9939f
 2021-01-22 23:16:21 +03:00
Leonid Yuriev
b1a2892038 mdbx: fix minor Coverity warnings. 
Change-Id: I0c5d23a36120b8d697887462a6ae86d66a435c3d
 2021-01-22 20:13:31 +03:00
Leonid Yuriev
b82d1d5063 mdbx: add MDBX_opt_spill_parent4child_denominator runtime option. 
More for https://github.com/erthink/libmdbx/issues/128

Change-Id: If1549c3855eb59c107eb115e6dbca515e81644f9
 2021-01-22 18:18:52 +03:00
Leonid Yuriev
56aaad03bc mdbx: add MDBX_opt_spill_max_denominator and MDBX_opt_spill_min_denominator runtime options. 
More for https://github.com/erthink/libmdbx/issues/128

Change-Id: I6b1b00e30ac11e9c2e3d3d3a29cc73079b41d539
 2021-01-22 17:25:44 +03:00
Leonid Yuriev
e0d4eaf819 mdbx: minor refine mdbx_txn_merge(). 2021-01-21 22:45:34 +03:00
Leonid Yuriev
1d58ad6d9b mdbx: fix Valgrind/ASAN controls. 
Change-Id: Idf526cbdb8ce8e43600efebbd1d12a8306a58184
 2021-01-21 22:45:28 +03:00
Leonid Yuriev
58e4eda1c6 mdbx: fix spilled page list memleak. 
Change-Id: I8b3b3664711943b0809be7bcc6580aafd4e4c286
 2021-01-21 22:45:28 +03:00
Leonid Yuriev
d26ae6875b mdbx: fix assertion to avoid false-positive UB detection by CLANG/LLVM. 
Assume this resolve https://github.com/erthink/libmdbx/issues/153

Change-Id: I02ceba1063a338b6dedf17629b51b3466358b0b8
 2021-01-21 22:45:28 +03:00
Leonid Yuriev
afe93137fc mdbx: fix minor assertions. 
Change-Id: Ib3e2dcb2cb419f41717351b2ac2264ffed906014
 2021-01-21 22:45:28 +03:00
Leonid Yuriev
bed14e60c2 mdbx-cmake: add -fsanitize-undefined-trap-on-error to compile flags and ENABLE_UBSAN to config.h. 
Change-Id: Ic18c0d59ccf32f9783a44de456893fbc8a0a7d32
 2021-01-21 22:45:23 +03:00
Leonid Yuriev
730b9ea3fd mdbx: add MDBX_PNL_ASCENDING build option and dependency for MDBX_DATA_MAGIC. 
Change-Id: I55283240b1dbbbc093927be5eaa5d42b87294f72
 2021-01-20 23:31:14 +03:00
Leonid Yuriev
6bd0277020 mdbx: minor fix mdbx_cursor_close(). 
Avoids reference to txn which could be already freed.

Change-Id: I4e1fadd9786ced1533bb8468d30e013455601228
 2021-01-20 09:55:11 +03:00
Leonid Yuriev
7d09de489d mdbx: fix minor MSVC warnings. 
Change-Id: I633b8847698dca25b6c395216744e157f6fb240d
 2021-01-20 09:55:11 +03:00
Leonid Yuriev
0d09c2cda9 mdbx: avoid overflow for dirty & leftover space inside mdbx_txn_info(). 
Change-Id: I36f4d62881e68f1bfe0d11fde2e849cdf1408ed2
 2021-01-20 02:07:44 +03:00
Leonid Yuriev
42670aa64a mdbx: fix MSVC warnings aroung size_t and unsigned conversion. 
Change-Id: I32bd3ee4fe02ac62cc121f33b2f84bf63c81cb75
 2021-01-20 01:39:21 +03:00
Leonid Yuriev
6d7daf6401 mdbx: fix audit for complex nested txn cases. 
Change-Id: I2419cbe0e8d3167e77ac99f397b97edb22717833
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
f28da7c9e3 mdbx: rework spilling, page-lists, commit & abort for nested transactions, etc (squashed). 
Change-Id: I09e4744586342aeae4efcfb71f41a3248ee47758
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
7bae2a7cd2 mdbx: provide MDBX_ENABLE_REFUND build option. 
Change-Id: I0e4c94a22f3b0957176431709604a7595f85252a
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
7183f62e13 mdbx: rework DBI_DIRTY and MDBX_TXN_DIRTY propagation. 
Change-Id: Ief12ee320ce7c7142632bd1142d847aac314db0b
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
4afc30c79f mdbx: rework mdbx_page_alloc(). 
Change-Id: I5c6e2ef23abf79096dd699f9dfbecf97bfc4fe37
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
c6a8f0f9af mdbx-tools: minor refine mdbx_chk error output. 
Change-Id: I4692a46c3e007fa2910ef67ec7fe21c7285e92eb
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
966d7027a3 mdbx: minor fix bad_page() debug output. 
Change-Id: I14531fdceaad054ad8c45bf9042fee73de1f9a36
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
dfc92d9f96 mdbx: minor fix mdbx_page_new() debug output. 
Change-Id: If60f972cc3271ca91feea923fb4f4be48d488b22
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
ac0f4e14a8 mdbx: minor fix mdbx_page_new() debug output. 
Change-Id: I2f386ec6276cab404607b5c3a79d8028d73dad62
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
ca597f4314 mdbx: fix mdbx_page_flush() for possibility of memleack. 
Change-Id: Ieab0c98fcd1020d49cce73234125ceb11a8f0b64
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
c1fef9b51d mdbx: fix mdbx_update_gc() for retired page list change case. 
Change-Id: Ic454262edcda8881182a15d6a521b90990e5551d
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
b8bb4b4cc4 mdbx: fix propagation child txn DBI-state to parent. 
Change-Id: I7d92fa99131a937af1a1a75b38af57bc6787ab37
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
4425fb0b83 mdbx: minor refine flags-reset inside mdbx_page_flush(). 
This produces little bit less code for most architectures.

Change-Id: I0440ec7b465ae2f12a8b09b0226f2c42bc52f7de
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
5bb254e629 mdbx: allow fastpath-commit for pure nested txns with parent-spilled pages. 
Change-Id: I471fd5e057c323e3701158761631e0a9999bf54a
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
8fce97dca9 mdbx: fix mdbx_cursors_eot(). 
Change-Id: I6021ff019f25110db96f9ef351d31a1f48b76c18
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
f6ff075335 mdbx: rework/speedup spill-pages list. 
Change-Id: I97e63f8d4c82ae682e898e9b04edceb0b01d82a5
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
d64b81c673 mdbx: remove senseless assertions from mdbx_refund_loose(). 
Change-Id: I028ae6bafb9358b61287fc0201c246baf160ca03
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
0ab263b329 mdbx: purge deleted spilled pagenums on refund. 
Change-Id: I325fbad82cbecb71c35dd1edd1a2abccb8108541
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
7bf147d8c2 mdbx: fix assertion inside mdbx_txn_commit_ex() for nested spilled txn. 
Change-Id: I54b37ea1b9591bdec9d8bd16d0f57280a1f9ee77
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
6ef7b2f588 mdbx: fix assertion inside mdbx_txn_begin_ex() for nested txn with spilled parent. 
Change-Id: Ie1f3cd3df6beb296a7bbdc72b39c1c4cb9105083
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
f0d79df9b9 mdbx: fix mdbx_page_flush(). 
Change-Id: I467c2367f2e99ae50996d0859ae22f70c9649a2d
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
05958a708e mdbx: fix minor/potential mem-page leak inside mdbx_page_unspill(). 
Change-Id: I4398a4d81eda494e4de9d62d81e3936c6357b744
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
21fbc36311 mdbx: minor refine mdbx_page_retire(). 
Change-Id: I29d20ed47f56e17dde8c9417bfa1ec95302c1966
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
cea29fe485 mdbx: fix assertion inside mdbx_page_retire(). 
Change-Id: I5d8f92b9fde1c961252757ea508403b5f51e7646
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
fe386a66df mdbx: minor refine mdbx_page_unspill(). 
Change-Id: I20bbd4c502a2258b945996fa1cf761eab7f88a80
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
92527a5206 mdbx: rename mdbx_page_spill to mdbx_cursor_spill(). 
Change-Id: I71628951d29ea39055b920a76521a19cb10ac8da
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
266bb70b7d mdbx: add debug to mdbx_kill_page(). 
Change-Id: Ifbb3164989a19d8ea48288b4eda1a8bcf31790ff
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
56758372cf mdbx: drop unused mdbx_audit(). 
Change-Id: I6d31c6ad83c058892319da6bbf38506b3f55085f
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
1314b29557 mdbx: add const for bsearch functions. 
Change-Id: Ie63eafa145bc8a63ae56b2693aaaf69bd21893aa
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
ef7814c018 mdbx: clean zero item of DPL-list. 
Change-Id: I7c60f3996783d97cee7cf2f0f9a7b856f4272641
 2021-01-19 23:53:16 +03:00
Leonid Yuriev
d50fff8410 mdbx: minor refine mdbx_page_check(). 
Change-Id: I325c20fe042635a0641cc24f5661408f3476139b
 2021-01-19 23:53:16 +03:00