Compare commits

..

52 Commits

Author SHA1 Message Date
Leonid Yuriev
113162b651 mdbx: release v0.11.1
Backward compatibility break:
-----------------------------

The database format signature has been changed to prevent
forward-interoperability with an previous releases, which may lead to a
[false positive diagnosis of database corruption](https://github.com/erthink/libmdbx/issues/238)
due to flaws of an old library versions.

This change is mostly invisible:

 - previously versions are unable to read/write a new DBs;
 - but the new release is able to handle an old DBs and will silently upgrade ones.

Acknowledgements:
-----------------
 - [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.

Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
2021-10-23 20:15:50 +03:00
Leonid Yuriev
5babf0872e mdbx++: add ifndef-guard for _CRT_SECURE_NO_WARNINGS. 2021-10-22 20:14:12 +03:00
Leonid Yuriev
331232858a adds updating db-format signature during DB open.
Fixes https://github.com/erthink/libmdbx/issues/238.
2021-10-22 20:13:57 +03:00
Leonid Yuriev
fcb8cd2145 mdbx: alter DB-format' signature and change version to v0.11.x (not a release).
Related to https://github.com/erthink/libmdbx/issues/238

Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
2021-10-21 15:17:18 +03:00
Leonid Yuriev
514910621e mdbx: return MDBX_CORRUPTED instead of MDBX_PAGE_NOTFOUND for invalid pages. 2021-10-15 01:11:20 +03:00
Leonid Yuriev
7251f47d5b mdbx: fix typo which lead any error conversion to the 1. 2021-10-14 20:03:37 +03:00
Leonid Yuriev
edda9515d6 mdbx: release v0.10.5 (hotfix).
Acknowledgements:
-----------------
 - [Noel Kuntze](https://github.com/Thermi) for immediately bug reporting.

Fixes:
------
 - Fixed unaligned access regression after the `#pragma pack` fix for modern compilers.
 - Added UBSAN-test to CI to avoid a regression(s) similar to lately fixed.
 - Fixed possibility of meta-pages clashing after manually turn to a particular meta-page using `mdbx_chk` utility.

Minors:
-------
 - Refined handling of weak or invalid meta-pages while a DB opening.
 - Refined providing information for the @MAIN and @GC sub-databases of a last committed modification transaction's ID.

Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
2021-10-13 16:35:26 +03:00
Leonid Yuriev
4632df5661 mdbx: cleanup non-persistent flags from meta-model. 2021-10-13 16:34:53 +03:00
Leonid Yuriev
11a5c50591 mdbx: fix turn_for_recovery() for possibility of meta-pages clashing after turn to a particular meta-page. 2021-10-13 16:34:53 +03:00
Leonid Yuriev
3092078709 mdbx: refine handling of weak or invalid meta-pages while a DB opening. 2021-10-13 16:34:53 +03:00
Leonid Yuriev
cbb71058ca mdbx: refine providing information for the @MAIN and @GC sub-databases of a last committed modification transaction's ID. 2021-10-13 16:34:25 +03:00
Leonid Yuriev
9dd15a4415 mdbx-ci: use Circle-CI for UBSAN-test pass. 2021-10-13 16:13:30 +03:00
Leonid Yuriev
30745e0621 mdbx: refix #pragma pack for modern compilers and aligned-required arches (hotfix).
Fix a regression after the https://github.com/erthink/libmdbx/issues/235
2021-10-13 16:13:22 +03:00
Leonid Yuriev
c9659e1aca mdbx: fix minor ChangeLog typo. 2021-10-11 17:18:06 +03:00
Leonid Yuriev
1fed51ac0d mdbx: update patch for buildroot (old versions). 2021-10-10 15:45:07 +03:00
Leonid Yuriev
590b225fcc mdbx: release v0.10.4
Acknowledgements:
-----------------
 - [Artem Vorotnikov](https://github.com/vorot93) for support [Rust wrapper](https://github.com/vorot93/mdbx-rs).
 - [Andrew Ashikhmin](https://github.com/yperbasis) for contributing to C++ API.

Fixes:
------
 - Fixed possibility of looping update GC during transaction commit (no public issue since the problem was discovered inside [Positive Technologies](https://www.ptsecurity.ru).
 - Fixed `#pragma pack` to avoid provoking some compilers to generate code with [unaligned access](https://github.com/erthink/libmdbx/issues/235).
 - Fixed `noexcept` for potentially throwing `txn::put()` of C++ API.

Minors:
-------
 - Added stochastic test script for checking small transactions cases.
 - Removed extra transaction commit/restart inside test framework.
 - In debugging builds fixed a too small (single page) by default DB shrink threshold.
2021-10-10 13:31:33 +03:00
Leonid Yuriev
2f8a429f91 mdbx: update ChangeLog. 2021-10-10 00:51:38 +03:00
Leonid Yuriev
64e6fa93fd mdbx: fix #pragma pack to avoid misalignment for some compilers.
Fixes https://github.com/erthink/libmdbx/issues/235.
2021-10-09 12:36:40 +03:00
Leonid Yuriev
ee917209fe mdbx-test: add stochastic-small script. 2021-10-09 12:30:39 +03:00
Leonid Yuriev
fa0a38c1ac mdbx: avoid single-page (too small) shrink threshold by default when MDBX_DEBUG > 0. 2021-10-09 12:30:39 +03:00
Leonid Yuriev
f936217309 mdbx-test: avoid extra transaction restart. 2021-10-09 12:30:35 +03:00
Leonid Yuriev
fe0ec8ceca mdbx: fix and refine mdbx_update_gc() to avoid infinite looping possibility (squashed). 2021-10-09 12:29:10 +03:00
Leonid Yuriev
6737d304a6 mdbx-ci: MDBX_FORCE_ASSERTION=1 for CI-build. 2021-10-09 12:29:05 +03:00
Leonid Yuriev
fe7186d549 mdbx: reflow comment (cosmetic). 2021-09-30 16:38:07 +03:00
Leonid Yuriev
c81226906a mdbx: update ChangeLog and Contributors list. 2021-09-28 00:37:33 +03:00
Leonid Yuriev
699361c5d0 mdbx: update bindings/wrappers list. 2021-09-27 20:39:19 +03:00
Leonid Yuriev
903bcd2466 mdbx-ci: switch github action from Ubuntu 16.04 to 18.04 2021-09-27 03:18:36 +03:00
yperbasis
c714ee9b55 mdbx++: remove noexcept from potentially throwing txn::put(). 2021-09-03 23:10:22 +03:00
Leonid Yuriev
bf603bdffc mdbx: release v0.10.3
Acknowledgements:
-----------------
 - [Francisco Vallarino](https://github.com/fjvallarino) for [Haskell bindings for libmdbx](https://hackage.haskell.org/package/libmdbx).
 - [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
 - [Andrea Lanfranchi](https://github.com/AndreaLanfranchi) for contributing.

Extensions and improvements:
----------------------------
 - Added `cursor::erase()` overloads for `key` and for `key-value`.
 - Resolve minor Coverity Scan issues (no fixes but some hint/comment were added).
 - Resolve minor UndefinedBehaviorSanitizer issues (no fixes but some workaround were added).

Fixes:
------
 - Always setup `madvise` while opening DB (fixes https://github.com/erthink/libmdbx/issues/231).
 - Fixed checking legacy `P_DIRTY` flag (`0x10`) for nested/sub-pages.

Minors:
-------
 - Fixed getting revision number from middle of history during amalgamation (GNU Makefile).
 - Fixed search GCC tools for LTO (CMake scripts).
 - Fixed/reorder dirs list for search CLANG tools for LTO (CMake scripts).
 - Fixed/workarounds for CLANG < 9.x
 - Fixed CMake warning about compatibility with 3.8.2
2021-08-27 22:47:12 +03:00
Leonid Yuriev
cd73caac1c mdbx-test: remove entropy source and use fully determined PRNG. 2021-08-27 15:03:59 +03:00
Leonid Yuriev
aec884f0d1 mdbx-make: extend cross-qemu target.
Change-Id: I889e53207904a8aaec8469165ba2de7574bf417b
2021-08-26 17:56:39 +03:00
Leonid Yuriev
e3ce6d645a mdbx-make: fix/filter-out cross-qemu target arch list.
Change-Id: Ibafe6217aefcbee9f7a14fa216cf331d2e56c7ce
2021-08-26 17:56:10 +03:00
Leonid Yuriev
d2cba98f70 mdbx: always setup madvise while opening DB.
This partially revert 7da64b725d.

Hope fix https://github.com/erthink/libmdbx/issues/231
2021-08-25 14:13:07 +03:00
Leonid Yuriev
217e951e68 mdbx-make: fix getting revision number from middle of history. 2021-08-25 13:24:22 +03:00
Leonid Yuriev
99b75b5004 mdbx: fix/model minor Coverity issues. 2021-08-16 23:45:56 +03:00
Leonid Yuriev
42d545e579 mdbx: fix minor zero-length memcmp() UB. 2021-08-14 17:46:34 +03:00
Leonid Yuriev
e3300259ff mdbx: add minor enum-related workarounds for UndefinedBeheviorSanitizer. 2021-08-14 16:43:16 +03:00
Leonid Yuriev
68273acc2a mdbx: add and use MDBX_NOSANITIZE_ENUM macro. 2021-08-14 16:43:12 +03:00
Leonid Yuriev
e20664fe55 mdbx: Merge branch 'master' into devel branch. 2021-08-14 16:01:16 +03:00
Leonid Yuriev
8c761f5774 mdbx: update Ethereum address. 2021-08-14 00:19:40 +03:00
Leonid Yuriev
b6ffec12e4 "mdbx: ignore legacy P_DIRTY flag (0x10) for nested/sub-pages. 2021-08-11 15:49:46 +03:00
Leonid Yuriev
5d4d02617d mdbx-cmake: fix search GCC tools for LTO. 2021-08-05 02:26:36 +03:00
Leonid Yuriev
327b283136 mdbx-cmake: fix/reorder dirs list for search CLANG tools for LTO. 2021-08-05 02:26:36 +03:00
Leonid Yuriev
4bb0c57e29 mdbx: minor fixes/workarounds for CLANG < 9.x 2021-08-04 15:43:32 +03:00
Leonid Yuriev
a9cae5e314 mdbx-cmake: avoid CMake warning about compatibility with 3.8.2 2021-08-04 14:09:47 +03:00
Leonid Yuriev
b9ac607a5e mdbx: update patch for buildroot (old versions). 2021-08-03 00:57:22 +03:00
Leonid Yuriev
7e035115bb mdbx-doc: add ref to Haskell bindings. 2021-08-02 09:05:50 +03:00
Leonid Yuriev
099dd68630 mdbx-doc: add a note about the need for git tags.
Related to https://github.com/erthink/libmdbx/issues/227.
2021-08-01 17:58:42 +03:00
Leonid Yuriev
5d7ef50054 mdbx-ci: disable RDP by default for AppVeyor. 2021-08-01 17:41:18 +03:00
Andrea Lanfranchi
2395564c17 mdbx++: add cursor::erase() overloads for key and for key-value.
Resolves https://github.com/erthink/libmdbx/pull/226
2021-07-27 01:27:57 +03:00
Andrea Lanfranchi
5e7a685ba8 mdbx: add basic CMakeSettings.json for Visual Studio.
* move build files out of project dir
2021-07-27 01:27:57 +03:00
Andrea Lanfranchi
f7cbf41f03 mdbx: add .vscode to gitignore. 2021-07-27 01:03:48 +03:00
36 changed files with 940 additions and 671 deletions

View File

@@ -11,8 +11,7 @@ jobs:
- TESTLOG: /tmp/test.log
steps:
- checkout
- run: make all
- run: ulimit -c unlimited && make check
- run: ulimit -c unlimited && MDBX_BUILD_OPTIONS="-DNDEBUG=1 -DMDBX_FORCE_ASSERTIONS=1" make test-ubsan
- run:
command: |
mkdir -p /tmp/artifacts

View File

@@ -3,4 +3,4 @@ freebsd_instance:
task:
install_script: pkg install -y gmake bash git
script: git fetch --tags --force && gmake check
script: git fetch --tags --force && gmake MDBX_BUILD_OPTIONS="-DNDEBUG=1 -DMDBX_FORCE_ASSERTIONS=1" check

2
.github/FUNDING.yml vendored
View File

@@ -9,4 +9,4 @@ community_bridge: # Replace with a single Community Bridge project-name e.g., cl
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
otechie: # Replace with a single Otechie username
custom: ['https://sobe.ru/na/libmdbx', 'https://paypal.me/erthink', 'https://etherscan.io/address/0xe8a741a2a0a4850c5bdbb415aaaaf8fb46f06fb7']
custom: ['https://sobe.ru/na/libmdbx', 'https://paypal.me/erthink', 'https://etherscan.io/address/0x19291d8658f762f3baceae1700c0b9466572ceab']

View File

@@ -8,3 +8,4 @@
^\.gitignore$
^\.travis\.yml$
^packages/buildroot/
^CMakeSettings\.json$

View File

@@ -61,6 +61,7 @@ ARMEL
ARMT
Artem
asan
Ashikhmin
asis
asm
asprintf
@@ -520,6 +521,7 @@ Firefox
firstvalue
fixedpoint
FIXME
fjvallarino
flagbit
flg
flipcoin
@@ -617,9 +619,11 @@ grep
gtags
gz
gzip
hackage
HAGL
hallvard
hardcoded
haskell
HASSEMAPHORE
hdiutil
hdr
@@ -705,6 +709,7 @@ inprocess
INSTEADOF
integerdup
integerkey
interoperability
interprocedural
intlimits
intptr
@@ -1112,6 +1117,7 @@ nordahead
NOREPLACE
NORESERVE
noreturn
NOSANITIZE
nospill
nosubdir
nosync
@@ -1791,6 +1797,7 @@ underfilled
UNDOC
unicode
UNIFORUM
uninit
uninstall
UNINSTALLING
uniq
@@ -1835,6 +1842,7 @@ Vaefnrs
valbool
valgrind
validator
Vallarino
valnum
valsize
valstr
@@ -1956,6 +1964,7 @@ xsize
yml
Yota
Yotta
yperbasis
Yq
yuriev
Zano

View File

@@ -27,11 +27,11 @@ jobs:
strategy:
matrix:
#, windows-latest
os: [ubuntu-latest, macos-latest, ubuntu-16.04]
os: [ubuntu-latest, macos-latest, ubuntu-18.04]
steps:
- uses: actions/checkout@v2
- name: fetch tags
run: git fetch --unshallow --tags --prune --force
- name: make check
run: make --keep-going all && MALLOC_CHECK_=7 MALLOC_PERTURB_=42 make --keep-going check
run: make MDBX_BUILD_OPTIONS="-DNDEBUG=1 -DMDBX_FORCE_ASSERTIONS=1" --keep-going all && MALLOC_CHECK_=7 MALLOC_PERTURB_=42 make MDBX_BUILD_OPTIONS="-DNDEBUG=1 -DMDBX_FORCE_ASSERTIONS=1" --keep-going check
shell: bash

1
.gitignore vendored
View File

@@ -15,6 +15,7 @@
.idea
.le.ini
.vs/
.vscode/
cmake-build-*
@*
core

View File

@@ -2,6 +2,7 @@ Contributors
============
- Alexey Naumov <alexey.naumov@gmail.com>
- Andrew Ashikhmin <andrey.ashikhmin@gmail.com>
- Chris Mikkelson <cmikk@qwest.net>
- Claude Brisson <claude.brisson@gmail.com>
- David Barbour <dmbarbour@gmail.com>
@@ -16,7 +17,7 @@ Contributors
- John Hewson <john@jahewson.com>
- Klaus Malorny <klaus.malorny@knipp.de>
- Kurt Zeilenga <kurt.zeilenga@isode.com>
- Leonid Yuriev <leo@yuriev.ru>, <lyuryev@ptsecurity.com>
- Leonid Yuriev <leo@yuriev.ru>, <lyuryev@ptsecurity.ru>
- Lorenz Bauer <lmb@cloudflare.com>
- Luke Yeager <lyeager@nvidia.com>
- Martin Hedenfalk <martin@bzero.se>

View File

@@ -34,10 +34,14 @@
## The Future will (be) Positive. Всё будет хорошо.
##
cmake_minimum_required(VERSION 3.8.2)
if(CMAKE_VERSION VERSION_LESS 3.12)
cmake_minimum_required(VERSION 3.8.2)
else()
cmake_minimum_required(VERSION 3.12)
endif()
cmake_policy(PUSH)
cmake_policy(VERSION 3.8.2)
cmake_policy(VERSION ${CMAKE_MINIMUM_REQUIRED_VERSION})
if(NOT CMAKE_VERSION VERSION_LESS 3.15)
cmake_policy(SET CMP0091 NEW)
endif()
@@ -228,8 +232,13 @@ else()
endif()
if(CMAKE_INTERPROCEDURAL_OPTIMIZATION_AVAILABLE
OR GCC_LTO_AVAILABLE OR MSVC_LTO_AVAILABLE OR CLANG_LTO_AVAILABLE)
OR GCC_LTO_AVAILABLE OR MSVC_LTO_AVAILABLE OR
(CLANG_LTO_AVAILABLE AND
((DEFINED MDBX_ENABLE_TESTS AND NOT MDBX_ENABLE_TESTS)
OR NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 4.0)))
option(INTERPROCEDURAL_OPTIMIZATION "Enable interprocedural/LTO optimization" ${INTERPROCEDURAL_OPTIMIZATION_DEFAULT})
else()
set(INTERPROCEDURAL_OPTIMIZATION OFF)
endif()
if(INTERPROCEDURAL_OPTIMIZATION)
@@ -328,9 +337,11 @@ list(FIND CMAKE_CXX_COMPILE_FEATURES cxx_std_20 HAS_CXX20)
if(NOT DEFINED MDBX_CXX_STANDARD)
if(DEFINED CMAKE_CXX_STANDARD)
set(MDBX_CXX_STANDARD ${CMAKE_CXX_STANDARD})
elseif(NOT HAS_CXX20 LESS 0)
elseif(NOT HAS_CXX20 LESS 0
AND NOT (CMAKE_COMPILER_IS_CLANG AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9))
set(MDBX_CXX_STANDARD 20)
elseif(NOT HAS_CXX17 LESS 0)
elseif(NOT HAS_CXX17 LESS 0
AND NOT (CMAKE_COMPILER_IS_CLANG AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 5))
set(MDBX_CXX_STANDARD 17)
elseif(NOT HAS_CXX14 LESS 0)
set(MDBX_CXX_STANDARD 14)
@@ -483,7 +494,7 @@ if(CMAKE_CXX_COMPILER_LOADED AND MDBX_CXX_STANDARD GREATER_EQUAL 11 AND MDBX_CXX
endif()
if(NOT MDBX_WITHOUT_MSVC_CRT
AND NOT (CMAKE_COMPILER_IS_GNUCXX AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 4.8)
AND NOT (CMAKE_COMPILER_IS_CLANG AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 4)
AND NOT (CMAKE_COMPILER_IS_CLANG AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 3.9)
AND NOT (MSVC AND MSVC_VERSION LESS 1900))
option(MDBX_BUILD_CXX "Build C++ portion" ON)
else()
@@ -560,7 +571,8 @@ macro(target_setup_options TARGET)
target_compile_options(${TARGET} INTERFACE "/Zc:__cplusplus")
endif()
endif()
if(CC_HAS_FASTMATH)
if(CC_HAS_FASTMATH
AND NOT (CMAKE_COMPILER_IS_CLANG AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 10))
target_compile_options(${TARGET} PRIVATE "-ffast-math")
endif()
if(CC_HAS_VISIBILITY)
@@ -583,7 +595,7 @@ macro(libmdbx_setup_libs TARGET MODE)
elseif(${CMAKE_SYSTEM_NAME} STREQUAL "Android")
target_link_libraries(${TARGET} ${MODE} log)
endif()
if(LIBCXX_FILESYSTEM AND MDBX_BUILD_CXX)
if(MDBX_CXX_STANDARD GREATER_EQUAL 17 AND LIBCXX_FILESYSTEM AND MDBX_BUILD_CXX)
target_link_libraries(${TARGET} ${MODE} ${LIBCXX_FILESYSTEM})
endif()
endmacro()

26
CMakeSettings.json Normal file
View File

@@ -0,0 +1,26 @@
{
"configurations": [
{
"name": "x64-Debug",
"generator": "Ninja",
"configurationType": "Debug",
"inheritEnvironments": [ "msvc_x64_x64" ],
"buildRoot": "${env.USERPROFILE}\\CMakeBuilds\\libmdbx\\build\\${name}",
"installRoot": "${env.USERPROFILE}\\CMakeBuilds\\libmdbx\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "",
"ctestCommandArgs": ""
},
{
"name": "x64-Release",
"generator": "Ninja",
"configurationType": "Release",
"inheritEnvironments": [ "msvc_x64_x64" ],
"buildRoot": "${env.USERPROFILE}\\CMakeBuilds\\libmdbx\\build\\${name}",
"installRoot": "${env.USERPROFILE}\\CMakeBuilds\\libmdbx\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "",
"ctestCommandArgs": ""
}
]
}

View File

@@ -1,8 +1,6 @@
ChangeLog
---------
## v0.10.3 (in development)
### TODO
- [Engage an "overlapped I/O" on Windows](https://github.com/erthink/libmdbx/issues/224).
@@ -16,10 +14,103 @@ ChangeLog
- [Support for RAW devices](https://github.com/erthink/libmdbx/issues/124).
- [Support MessagePack for Keys & Values](https://github.com/erthink/libmdbx/issues/115).
- [Engage new terminology](https://github.com/erthink/libmdbx/issues/137).
- Finalize C++ API (few typos and trivia bugs are still likely for now).
- Finalize C++ API (few typos and bugs are still maybe for now).
- Packages for [Astra Linux](https://astralinux.ru/), [ALT Linux](https://www.altlinux.org/), [ROSA Linux](https://www.rosalinux.ru/), etc.
## v0.11.1 at 2021-10-23
### Backward compatibility break:
The database format signature has been changed to prevent
forward-interoperability with an previous releases, which may lead to a
[false positive diagnosis of database corruption](https://github.com/erthink/libmdbx/issues/238)
due to flaws of an old library versions.
This change is mostly invisible:
- previously versions are unable to read/write a new DBs;
- but the new release is able to handle an old DBs and will silently upgrade ones.
Acknowledgements:
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
## v0.10.5 at 2021-10-13 (obsolete, please use v0.11.1)
Unfortunately, the `v0.10.5` accidentally comes not full-compatible with previous releases:
- `v0.10.5` can read/processing DBs created by previous releases, i.e. the backward-compatibility is provided;
- however, previous releases may lead to false-corrupted state with DB that was touched by `v0.10.5`, i.e. the forward-compatibility is broken for `v0.10.4` and earlier.
This cannot be fixed, as it requires fixing past versions, which as a result we will just get a current version.
Therefore, it is recommended to use `v0.11.1` instead of `v0.10.5`.
Acknowledgements:
- [Noel Kuntze](https://github.com/Thermi) for immediately bug reporting.
Fixes:
- Fixed unaligned access regression after the `#pragma pack` fix for modern compilers.
- Added UBSAN-test to CI to avoid a regression(s) similar to lately fixed.
- Fixed possibility of meta-pages clashing after manually turn to a particular meta-page using `mdbx_chk` utility.
Minors:
- Refined handling of weak or invalid meta-pages while a DB opening.
- Refined providing information for the @MAIN and @GC sub-databases of a last committed modification transaction's ID.
## v0.10.4 at 2021-10-10
Acknowledgements:
- [Artem Vorotnikov](https://github.com/vorot93) for support [Rust wrapper](https://github.com/vorot93/mdbx-rs).
- [Andrew Ashikhmin](https://github.com/yperbasis) for contributing to C++ API.
Fixes:
- Fixed possibility of looping update GC during transaction commit (no public issue since the problem was discovered inside [Positive Technologies](https://www.ptsecurity.ru)).
- Fixed `#pragma pack` to avoid provoking some compilers to generate code with [unaligned access](https://github.com/erthink/libmdbx/issues/235).
- Fixed `noexcept` for potentially throwing `txn::put()` of C++ API.
Minors:
- Added stochastic test script for checking small transactions cases.
- Removed extra transaction commit/restart inside test framework.
- In debugging builds fixed a too small (single page) by default DB shrink threshold.
## v0.10.3 at 2021-08-27
Acknowledgements:
- [Francisco Vallarino](https://github.com/fjvallarino) for [Haskell bindings for libmdbx](https://hackage.haskell.org/package/libmdbx).
- [Alex Sharov](https://github.com/AskAlexSharov) for reporting and testing.
- [Andrea Lanfranchi](https://github.com/AndreaLanfranchi) for contributing.
Extensions and improvements:
- Added `cursor::erase()` overloads for `key` and for `key-value`.
- Resolve minor Coverity Scan issues (no fixes but some hint/comment were added).
- Resolve minor UndefinedBehaviorSanitizer issues (no fixes but some workaround were added).
Fixes:
- Always setup `madvise` while opening DB (fixes https://github.com/erthink/libmdbx/issues/231).
- Fixed checking legacy `P_DIRTY` flag (`0x10`) for nested/sub-pages.
Minors:
- Fixed getting revision number from middle of history during amalgamation (GNU Makefile).
- Fixed search GCC tools for LTO (CMake scripts).
- Fixed/reorder dirs list for search CLANG tools for LTO (CMake scripts).
- Fixed/workarounds for CLANG < 9.x
- Fixed CMake warning about compatibility with 3.8.2
## v0.10.2 at 2021-07-26
Acknowledgements:

View File

@@ -292,7 +292,7 @@ MAN_SRCDIR := src/man1/
ALLOY_DEPS := $(shell git ls-files src/)
git_DIR := $(shell if [ -d .git ]; then echo .git; elif [ -s .git -a -f .git ]; then grep '^gitdir: ' .git | cut -d ':' -f 2; else echo git_directory_is_absent; fi)
MDBX_GIT_VERSION = $(shell set -o pipefail; git describe --tags 2>&- | sed -n 's|^v*\([0-9]\{1,\}\.[0-9]\{1,\}\.[0-9]\{1,\}\)\(.*\)|\1|p' || echo 'Please fetch tags and/or use non-obsolete git version')
MDBX_GIT_REVISION = $(shell set -o pipefail; git rev-list --count HEAD ^`git tag --sort=-version:refname 2>&- | sed -n '/^\(v[0-9]\+\.[0-9]\+\.[0-9]\+\)*/p;q' || echo 'git_tag_with_sort_was_failed'` 2>&- || echo 'Please use non-obsolete git version')
MDBX_GIT_REVISION = $(shell set -o pipefail; git rev-list `git describe --tags --abbrev=0`..HEAD --count 2>&- || echo 'Please fetch tags and/or use non-obsolete git version')
MDBX_GIT_TIMESTAMP = $(shell git show --no-patch --format=%cI HEAD 2>&- || echo 'Please install latest get version')
MDBX_GIT_DESCRIBE = $(shell git describe --tags --long --dirty=-dirty 2>&- || echo 'Please fetch tags and/or install non-obsolete git version')
MDBX_VERSION_SUFFIX = $(shell set -o pipefail; echo -n '$(MDBX_GIT_DESCRIBE)' | tr -c -s '[a-zA-Z0-9]' _)
@@ -586,7 +586,7 @@ endif
################################################################################
# Cross-compilation simple test
CROSS_LIST = sparc64-linux-gnu-gcc alpha-linux-gnu-gcc mips-linux-gnu-gcc \
CROSS_LIST = mips-linux-gnu-gcc \
powerpc64-linux-gnu-gcc powerpc-linux-gnu-gcc \
arm-linux-gnueabihf-gcc aarch64-linux-gnu-gcc \
sh4-linux-gnu-gcc mips64-linux-gnuabi64-gcc \
@@ -597,7 +597,7 @@ CROSS_LIST = sparc64-linux-gnu-gcc alpha-linux-gnu-gcc mips-linux-gnu-gcc \
# s390x-linux-gnu-gcc - works (previously: qemu hang/abort)
# sparc64-linux-gnu-gcc - coredump (qemu mmap-troubles, previously: qemu fails fcntl for F_SETLK/F_GETLK)
# alpha-linux-gnu-gcc - coredump (qemu mmap-troubles)
CROSS_LIST_NOQEMU =
CROSS_LIST_NOQEMU = sparc64-linux-gnu-gcc alpha-linux-gnu-gcc
cross-gcc:
@echo ' Re-building by cross-compiler for: $(CROSS_LIST_NOQEMU) $(CROSS_LIST)'
@@ -620,7 +620,7 @@ cross-qemu:
echo "===================== $$CC + qemu"; \
$(MAKE) IOARENA=false CXXSTD= clean && \
CC=$$CC CXX=$$(echo $$CC | sed 's/-gcc/-g++/') EXE_LDFLAGS=-static MDBX_BUILD_OPTIONS="-DMDBX_SAFE4QEMU $(MDBX_BUILD_OPTIONS)" \
$(MAKE) IOARENA=false test-singleprocess || exit $$?; \
$(MAKE) IOARENA=false smoke-singleprocess test-singleprocess || exit $$?; \
done
#< dist-cutoff-end

View File

@@ -385,6 +385,12 @@ target platform. Obviously you need building tools itself, i.e. `git`,
and `make options` are also available for listing existing targets
and build options respectively.
The only significant specificity is that git' tags are required
to build from complete (not amalgamated) source codes.
Executing **`git fetch --tags --force --prune`** is enough to get ones,
or `git fetch --unshallow --tags --prune --force` after the Github's
[`actions/checkout@v2`](https://github.com/actions/checkout) either set **`fetch-depth: 0`** for it.
So just using CMake or GNU Make in your habitual manner and feel free to
fill an issue or make pull request in the case something will be
unexpected or broken down.
@@ -558,15 +564,16 @@ Bindings
| Runtime | Repo | Author |
| ------- | ------ | ------ |
| Haskell | [libmdbx-hs](https://hackage.haskell.org/package/libmdbx) | [Francisco Vallarino](https://github.com/fjvallarino) |
| Python (draft) | [python-bindings](https://github.com/erthink/libmdbx/commits/python-bindings) branch | [Noel Kuntze](https://github.com/Thermi)
| NodeJS | [lmdbx-store](https://github.com/kriszyp/lmdbx-store) | [Kris Zyp](https://github.com/kriszyp/)
| NodeJS | [node-mdbx](https://www.npmjs.com/package/node-mdbx/) | [Сергей Федотов](mailto:sergey.fedotov@corp.mail.ru) |
| Ruby | [ruby-mdbx](https://rubygems.org/gems/mdbx/) | [Mahlon E. Smith](https://github.com/mahlonsmith) |
| Go | [mdbx-go](https://github.com/torquem-ch/mdbx-go) | [Alex Sharov](https://github.com/AskAlexSharov) |
| [Nim](https://en.wikipedia.org/wiki/Nim_(programming_language)) | [NimDBX](https://github.com/snej/nimdbx) | [Jens Alfke](https://github.com/snej)
| Rust | [heed](https://github.com/Kerollmops/heed), [mdbx-rs](https://github.com/Kerollmops/mdbx-rs) | [Clément Renault](https://github.com/Kerollmops) |
| Java | [mdbxjni](https://github.com/castortech/mdbxjni) | [Castor Technologies](https://castortech.com/) |
| .NET | [mdbx.NET](https://github.com/wangjia184/mdbx.NET) | [Jerry Wang](https://github.com/wangjia184) |
| Rust | [mdbx-rs](https://github.com/vorot93/mdbx-rs) | [Artem Vorotnikov](https://github.com/vorot93) |
| Java (obsolete) | [mdbxjni](https://github.com/castortech/mdbxjni) | [Castor Technologies](https://castortech.com/) |
| .NET (obsolete) | [mdbx.NET](https://github.com/wangjia184/mdbx.NET) | [Jerry Wang](https://github.com/wangjia184) |
<!-- section-end -->

View File

@@ -1,4 +1,4 @@
version: 0.10.2.{build}
version: 0.11.1.{build}
environment:
matrix:
@@ -52,8 +52,8 @@ matrix:
configuration: Release
# Enable RDP for troubleshooting
init:
- ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))
#init:
# - ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))
before_build:
- git clean -x -f -d

View File

@@ -13,9 +13,14 @@
## limitations under the License.
##
cmake_minimum_required(VERSION 3.8.2)
if(CMAKE_VERSION VERSION_LESS 3.12)
cmake_minimum_required(VERSION 3.8.2)
else()
cmake_minimum_required(VERSION 3.12)
endif()
cmake_policy(PUSH)
cmake_policy(VERSION 3.8.2)
cmake_policy(VERSION ${CMAKE_MINIMUM_REQUIRED_VERSION})
if(CMAKE_VERSION MATCHES ".*MSVC.*" AND CMAKE_VERSION VERSION_LESS 3.16)
message(FATAL_ERROR "CMake from MSVC kit is unfit! "
@@ -286,21 +291,21 @@ if(CMAKE_COMPILER_IS_GNU${CMAKE_PRIMARY_LANG})
set(gcc_suffix "")
if(gcc_collect_valid AND gcc_collect)
string(REGEX MATCH "^(.*cc)(-.+)$" gcc_suffix_valid ${gcc_collect})
string(REGEX MATCH "^(.*(cc|\\+\\+))(-.+)$" gcc_suffix_valid ${gcc_collect})
if(gcc_suffix_valid)
string(REGEX MATCH "^(.*cc)(-.+)$" "\\2" gcc_suffix ${gcc_collect})
string(REGEX REPLACE "^(.*(cc|\\+\\+))(-.+)$" "\\3" gcc_suffix ${gcc_collect})
endif()
endif()
get_filename_component(gcc_dir ${CMAKE_${CMAKE_PRIMARY_LANG}_COMPILER} DIRECTORY)
if(NOT CMAKE_GCC_AR)
find_program(CMAKE_GCC_AR NAMES gcc${gcc_suffix}-ar gcc-ar${gcc_suffix} PATHS ${gcc_dir} NO_DEFAULT_PATH)
find_program(CMAKE_GCC_AR NAMES "gcc${gcc_suffix}-ar" "gcc-ar${gcc_suffix}" PATHS "${gcc_dir}" NO_DEFAULT_PATH)
endif()
if(NOT CMAKE_GCC_NM)
find_program(CMAKE_GCC_NM NAMES gcc${gcc_suffix}-nm gcc-nm${gcc_suffix} PATHS ${gcc_dir} NO_DEFAULT_PATH)
find_program(CMAKE_GCC_NM NAMES "gcc${gcc_suffix}-nm" "gcc-nm${gcc_suffix}" PATHS "${gcc_dir}" NO_DEFAULT_PATH)
endif()
if(NOT CMAKE_GCC_RANLIB)
find_program(CMAKE_GCC_RANLIB NAMES gcc${gcc_suffix}-ranlib gcc-ranlib${gcc_suffix} PATHS ${gcc_dir} NO_DEFAULT_PATH)
find_program(CMAKE_GCC_RANLIB NAMES "gcc${gcc_suffix}-ranlib" "gcc-ranlib${gcc_suffix}" PATHS "${gcc_dir}" NO_DEFAULT_PATH)
endif()
unset(gcc_dir)
@@ -342,30 +347,40 @@ if(CMAKE_COMPILER_IS_CLANG)
OUTPUT_VARIABLE clang_search_dirs RESULT_VARIABLE clang_probe_result ERROR_QUIET)
unset(clang_bindirs)
unset(clang_bindirs_x)
unset(clang_libdirs)
unset(clang_libdirs_x)
if(clang_probe_result EQUAL 0)
string(REGEX MATCH "(^|\n.*)(.*programs: =)([^\n]+)((\n.*)|$)" regexp_valid ${clang_search_dirs})
if(regexp_valid)
string(REGEX REPLACE "(^|\n.*)(.*programs: =)([^\n]+)((\n.*)|$)" "\\3" list ${clang_search_dirs})
string(REPLACE ":" ";" list "${list}")
#set(clang_bindirs "")
foreach(dir IN LISTS list)
get_filename_component(dir "${dir}" REALPATH)
string(REGEX MATCH "(^|\n.*)(.*programs: =)([^\n]+)((\n.*)|$)" regexp_valid ${clang_search_dirs})
if(regexp_valid)
string(REGEX REPLACE "(^|\n.*)(.*programs: =)([^\n]+)((\n.*)|$)" "\\3" list ${clang_search_dirs})
string(REPLACE ":" ";" list "${list}")
foreach(dir IN LISTS list)
get_filename_component(dir "${dir}" REALPATH)
if(dir MATCHES ".*llvm.*" OR dir MATCHES ".*clang.*")
list(APPEND clang_bindirs "${dir}")
endforeach()
list(REMOVE_DUPLICATES clang_bindirs)
endif()
string(REGEX MATCH "(^|\n.*)(.*libraries: =)([^\n]+)((\n.*)|$)" regexp_valid ${clang_search_dirs})
if(regexp_valid)
string(REGEX REPLACE "(^|\n.*)(.*libraries: =)([^\n]+)((\n.*)|$)" "\\3" list ${clang_search_dirs})
string(REPLACE ":" ";" list "${list}")
#set(clang_libdirs "")
foreach(dir IN LISTS list)
get_filename_component(dir "${dir}" REALPATH)
else()
list(APPEND clang_bindirs_x "${dir}")
endif()
endforeach()
list(APPEND clang_bindirs "${clang_bindirs_x}")
list(REMOVE_DUPLICATES clang_bindirs)
endif()
string(REGEX MATCH "(^|\n.*)(.*libraries: =)([^\n]+)((\n.*)|$)" regexp_valid ${clang_search_dirs})
if(regexp_valid)
string(REGEX REPLACE "(^|\n.*)(.*libraries: =)([^\n]+)((\n.*)|$)" "\\3" list ${clang_search_dirs})
string(REPLACE ":" ";" list "${list}")
foreach(dir IN LISTS list)
get_filename_component(dir "${dir}" REALPATH)
if(dir MATCHES ".*llvm.*" OR dir MATCHES ".*clang.*")
list(APPEND clang_libdirs "${dir}")
endforeach()
list(REMOVE_DUPLICATES clang_libdirs)
endif()
else()
list(APPEND clang_libdirs_x "${dir}")
endif()
endforeach()
list(APPEND clang_libdirs "${clang_libdirs_x}")
list(REMOVE_DUPLICATES clang_libdirs)
endif()
else()
get_filename_component(clang_bindirs ${CMAKE_${CMAKE_PRIMARY_LANG}_COMPILER} DIRECTORY)
if(CMAKE_HOST_SYSTEM_NAME STREQUAL "Windows")

View File

@@ -13,9 +13,14 @@
## limitations under the License.
##
cmake_minimum_required(VERSION 3.8.2)
if(CMAKE_VERSION VERSION_LESS 3.12)
cmake_minimum_required(VERSION 3.8.2)
else()
cmake_minimum_required(VERSION 3.12)
endif()
cmake_policy(PUSH)
cmake_policy(VERSION 3.8.2)
cmake_policy(VERSION ${CMAKE_MINIMUM_REQUIRED_VERSION})
include(CheckLibraryExists)
check_library_exists(gcov __gcov_flush "" HAVE_GCOV)

View File

@@ -13,9 +13,14 @@
## limitations under the License.
##
cmake_minimum_required(VERSION 3.8.2)
if(CMAKE_VERSION VERSION_LESS 3.12)
cmake_minimum_required(VERSION 3.8.2)
else()
cmake_minimum_required(VERSION 3.12)
endif()
cmake_policy(PUSH)
cmake_policy(VERSION 3.8.2)
cmake_policy(VERSION ${CMAKE_MINIMUM_REQUIRED_VERSION})
macro(add_compile_flags languages)
foreach(_lang ${languages})

54
mdbx.h
View File

@@ -461,7 +461,8 @@ typedef mode_t mdbx_mode_t;
#endif /* MDBX_PRINTF_ARGS */
#if defined(DOXYGEN) || \
(defined(__cplusplus) && __has_cpp_attribute(maybe_unused) && \
(defined(__cplusplus) && __cplusplus >= 201603 && \
__has_cpp_attribute(maybe_unused) && \
__has_cpp_attribute(maybe_unused) >= 201603) || \
(!defined(__cplusplus) && defined(__STDC_VERSION__) && \
__STDC_VERSION__ > 202005L)
@@ -472,6 +473,12 @@ typedef mode_t mdbx_mode_t;
#define MDBX_MAYBE_UNUSED
#endif /* MDBX_MAYBE_UNUSED */
#if __has_attribute(no_sanitize)
#define MDBX_NOSANITIZE_ENUM __attribute((__no_sanitize__("enum")))
#else
#define MDBX_NOSANITIZE_ENUM
#endif /* MDBX_NOSANITIZE_ENUM */
/* Oh, below are some songs and dances since:
* - C++ requires explicit definition of the necessary operators.
* - the proper implementation of DEFINE_ENUM_FLAG_OPERATORS for C++ required
@@ -497,28 +504,40 @@ typedef mode_t mdbx_mode_t;
/// used to define flags (based on Microsoft's DEFINE_ENUM_FLAG_OPERATORS).
#define DEFINE_ENUM_FLAG_OPERATORS(ENUM) \
extern "C++" { \
MDBX_CXX01_CONSTEXPR ENUM operator|(ENUM a, ENUM b) { \
MDBX_NOSANITIZE_ENUM MDBX_CXX01_CONSTEXPR ENUM operator|(ENUM a, ENUM b) { \
return ENUM(unsigned(a) | unsigned(b)); \
} \
MDBX_CXX14_CONSTEXPR ENUM &operator|=(ENUM &a, ENUM b) { return a = a | b; } \
MDBX_CXX01_CONSTEXPR ENUM operator&(ENUM a, ENUM b) { \
MDBX_NOSANITIZE_ENUM MDBX_CXX14_CONSTEXPR ENUM &operator|=(ENUM &a, \
ENUM b) { \
return a = a | b; \
} \
MDBX_NOSANITIZE_ENUM MDBX_CXX01_CONSTEXPR ENUM operator&(ENUM a, ENUM b) { \
return ENUM(unsigned(a) & unsigned(b)); \
} \
MDBX_CXX01_CONSTEXPR ENUM operator&(ENUM a, unsigned b) { \
MDBX_NOSANITIZE_ENUM MDBX_CXX01_CONSTEXPR ENUM operator&(ENUM a, \
unsigned b) { \
return ENUM(unsigned(a) & b); \
} \
MDBX_CXX01_CONSTEXPR ENUM operator&(unsigned a, ENUM b) { \
MDBX_NOSANITIZE_ENUM MDBX_CXX01_CONSTEXPR ENUM operator&(unsigned a, \
ENUM b) { \
return ENUM(a & unsigned(b)); \
} \
MDBX_CXX14_CONSTEXPR ENUM &operator&=(ENUM &a, ENUM b) { return a = a & b; } \
MDBX_CXX14_CONSTEXPR ENUM &operator&=(ENUM &a, unsigned b) { \
MDBX_NOSANITIZE_ENUM MDBX_CXX14_CONSTEXPR ENUM &operator&=(ENUM &a, \
ENUM b) { \
return a = a & b; \
} \
MDBX_NOSANITIZE_ENUM MDBX_CXX14_CONSTEXPR ENUM &operator&=(ENUM &a, \
unsigned b) { \
return a = a & b; \
} \
MDBX_CXX01_CONSTEXPR unsigned operator~(ENUM a) { return ~unsigned(a); } \
MDBX_CXX01_CONSTEXPR ENUM operator^(ENUM a, ENUM b) { \
MDBX_NOSANITIZE_ENUM MDBX_CXX01_CONSTEXPR ENUM operator^(ENUM a, ENUM b) { \
return ENUM(unsigned(a) ^ unsigned(b)); \
} \
MDBX_CXX14_CONSTEXPR ENUM &operator^=(ENUM &a, ENUM b) { return a = a ^ b; } \
MDBX_NOSANITIZE_ENUM MDBX_CXX14_CONSTEXPR ENUM &operator^=(ENUM &a, \
ENUM b) { \
return a = a ^ b; \
} \
}
#else /* __cplusplus */
/* nope for C since it always allows these operators for enums */
@@ -549,9 +568,9 @@ typedef mode_t mdbx_mode_t;
extern "C" {
#endif
/* MDBX version 0.10.x */
/* MDBX version 0.11.x */
#define MDBX_VERSION_MAJOR 0
#define MDBX_VERSION_MINOR 10
#define MDBX_VERSION_MINOR 11
#ifndef LIBMDBX_API
#if defined(LIBMDBX_EXPORTS)
@@ -790,6 +809,10 @@ enum MDBX_log_level_t {
and all other log-messages */
MDBX_LOG_EXTRA = 7,
#ifdef ENABLE_UBSAN
MDBX_LOG_MAX = 7 /* avoid UBSAN false-positive trap by a tests */,
#endif /* ENABLE_UBSAN */
/** for \ref mdbx_setup_debug() only: Don't change current settings */
MDBX_LOG_DONTCHANGE = -1
};
@@ -803,6 +826,8 @@ typedef enum MDBX_log_level_t MDBX_log_level_t;
* effect, but `MDBX_DBG_ASSERT`, `MDBX_DBG_AUDIT` and `MDBX_DBG_JITTER` only if
* libmdbx builded with \ref MDBX_DEBUG. */
enum MDBX_debug_flags_t {
MDBX_DBG_NONE = 0,
/** Enable assertion checks.
* Requires build with \ref MDBX_DEBUG > 0 */
MDBX_DBG_ASSERT = 1,
@@ -825,6 +850,11 @@ enum MDBX_debug_flags_t {
/** Allow read and write transactions overlapping for the same thread */
MDBX_DBG_LEGACY_OVERLAP = 32,
#ifdef ENABLE_UBSAN
MDBX_DBG_MAX = ((unsigned)MDBX_LOG_MAX) << 16 |
63 /* avoid UBSAN false-positive trap by a tests */,
#endif /* ENABLE_UBSAN */
/** for mdbx_setup_debug() only: Don't change current settings */
MDBX_DBG_DONTCHANGE = -1
};

View File

@@ -3606,8 +3606,7 @@ public:
inline MDBX_error_t put(map_handle map, const slice &key, slice *value,
MDBX_put_flags_t flags) noexcept;
inline void put(map_handle map, const slice &key, slice value,
put_mode mode) noexcept;
inline void put(map_handle map, const slice &key, slice value, put_mode mode);
inline void insert(map_handle map, const slice &key, slice value);
inline value_result try_insert(map_handle map, const slice &key, slice value);
inline slice insert_reserve(map_handle map, const slice &key,
@@ -3626,6 +3625,7 @@ public:
inline value_result try_update_reserve(map_handle map, const slice &key,
size_t value_length);
/// \brief Removes all values for given key.
inline bool erase(map_handle map, const slice &key);
/// \brief Removes the particular multi-value entry of the key.
@@ -3864,7 +3864,18 @@ public:
inline slice update_reserve(const slice &key, size_t value_length);
inline value_result try_update_reserve(const slice &key, size_t value_length);
/// \brief Removes single key-value pair or all multi-values at the current
/// cursor position.
inline bool erase(bool whole_multivalue = false);
/// \brief Seeks and removes first value or whole multi-value of the given
/// key.
/// \return `True` if the key is found and a value(s) is removed.
inline bool erase(const slice &key, bool whole_multivalue = true);
/// \brief Seeks and removes the particular multi-value entry of the key.
/// \return `True` if the given key-value pair is found and removed.
inline bool erase(const slice &key, const slice &value);
};
/// \brief Managed cursor.
@@ -5154,7 +5165,7 @@ inline MDBX_error_t txn::put(map_handle map, const slice &key, slice *value,
}
inline void txn::put(map_handle map, const slice &key, slice value,
put_mode mode) noexcept {
put_mode mode) {
error::success_or_throw(put(map, key, &value, MDBX_put_flags_t(mode)));
}
@@ -5740,6 +5751,16 @@ inline bool cursor::erase(bool whole_multivalue) {
}
}
inline bool cursor::erase(const slice &key, bool whole_multivalue) {
bool found = seek(key);
return found ? erase(whole_multivalue) : found;
}
inline bool cursor::erase(const slice &key, const slice &value) {
move_result data = find_multivalue(key, value, false);
return data.done ? erase() : data.done;
}
} // namespace mdbx
//------------------------------------------------------------------------------

View File

@@ -1,10 +1,9 @@
From 7834ae8fc834f5f7b98d45703079cd94e5402ed6 Mon Sep 17 00:00:00 2001
From ce7b70a572cf77d8f37e5047d5d9c8361b2675ae Mon Sep 17 00:00:00 2001
From: Leonid Yuriev <leo@yuriev.ru>
Date: Fri, 27 Nov 2020 16:31:12 +0300
Cc: Heiko Thiery <heiko.thiery@gmail.com>, Thomas Petazzoni <thomas.petazzoni@bootlin.com>, Leonid Yuriev <leo@yuriev.ru
Subject: [PATCH v5 1/1] package/libmdbx: new package (library/database).
Date: Sun, 10 Oct 2021 15:31:40 +0300
Subject: [PATCH] package/libmdbx: new package (library/database).
This patch adds libmdbx v0.9.2:
This patch adds libmdbx v0.10.4:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities,
focused on creating unique lightweight solutions.
@@ -13,44 +12,14 @@ This patch adds libmdbx v0.9.2:
- https://github.com/erthink/libmdbx
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)
Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch for `pthread_yield()`.
- passed `utils/check-package package/libmdbx/*` (suggested by Heiko Thiery)
- passed `utils/test-pkg -a -p libmdbx`,
except w/o threads & w/o MMU (suggested by Heiko Thiery).
Changes v3 -> v4:
- fix passing BR2_PACKAGE_LIBMDBX_TOOLS option to cmake.
- fix using `depend on` instead of `select`,
and add `comment` for C++ toolchain (suggested by Heiko Thiery).
- fix minor help typo.
Changes v4 -> v5:
- libmdbx version v0.9.1 -> v0.9.2 (released 2020-11-27)
- dropped all patch since not needed.
- cosmetic changes (suggested by Thomas Petazzoni <thomas.petazzoni@bootlin.com>).
- added dependence of BR2_TOOLCHAIN_HAS_SYNC_4 (suggested by Thomas Petazzoni <thomas.petazzoni@bootlin.com>).
- added dependence of !BR2_TOOLCHAIN_HAS_GCC_BUG_64735 (suggested by Thomas Petazzoni <thomas.petazzoni@bootlin.com>).
- take in account the BR2_SHARED_STATIC_LIBS (suggested by Thomas Petazzoni <thomas.petazzoni@bootlin.com>).
Signed-off-by: Leonid Yuriev <leo@yuriev.ru>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
DEVELOPERS | 3 +++
package/Config.in | 1 +
package/libmdbx/Config.in | 45 ++++++++++++++++++++++++++++++++++++
package/libmdbx/libmdbx.hash | 5 ++++
package/libmdbx/libmdbx.mk | 33 ++++++++++++++++++++++++++
5 files changed, 87 insertions(+)
package/libmdbx/libmdbx.mk | 42 +++++++++++++++++++++++++++++++++
5 files changed, 96 insertions(+)
create mode 100644 package/libmdbx/Config.in
create mode 100644 package/libmdbx/libmdbx.hash
create mode 100644 package/libmdbx/libmdbx.mk
@@ -134,28 +103,28 @@ index 0000000000..d13f73938f
+ !BR2_TOOLCHAIN_GCC_AT_LEAST_4_4
diff --git a/package/libmdbx/libmdbx.hash b/package/libmdbx/libmdbx.hash
new file mode 100644
index 0000000000..0d3501f1d9
index 0000000000..326cf57bb6
--- /dev/null
+++ b/package/libmdbx/libmdbx.hash
@@ -0,0 +1,5 @@
+# Hashes from: https://github.com/erthink/libmdbx/releases/
+sha256 c35cc53d66d74ebfc86e39441ba26276541ac7892bf91dba1e70c83665a02767 libmdbx-amalgamated-0.9.2.tar.gz
+sha256 e11d5339a1e1cc34407898933b62a208936fd761a2cc31e11244d581d1d2b5d0 libmdbx-amalgamated-0.10.4.tar.gz
+
+# Locally calculated
+sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE
diff --git a/package/libmdbx/libmdbx.mk b/package/libmdbx/libmdbx.mk
new file mode 100644
index 0000000000..f3720130ec
index 0000000000..f38e4a533b
--- /dev/null
+++ b/package/libmdbx/libmdbx.mk
@@ -0,0 +1,33 @@
@@ -0,0 +1,42 @@
+################################################################################
+#
+# libmdbx
+#
+################################################################################
+
+LIBMDBX_VERSION = 0.9.2
+LIBMDBX_VERSION = 0.10.4
+LIBMDBX_SOURCE = libmdbx-amalgamated-$(LIBMDBX_VERSION).tar.gz
+LIBMDBX_SITE = https://github.com/erthink/libmdbx/releases/download/v$(LIBMDBX_VERSION)
+LIBMDBX_SUPPORTS_IN_SOURCE_BUILD = NO
@@ -165,7 +134,12 @@ index 0000000000..f3720130ec
+LIBMDBX_STRIP_COMPONENTS = 0
+LIBMDBX_INSTALL_STAGING = YES
+
+LIBMDBX_CONF_OPTS = -DMDBX_INSTALL_MANPAGES=OFF -DBUILD_FOR_NATIVE_CPU=OFF \
+# Set CMAKE_BUILD_TYPE to Release to remove -Werror and avoid a build failure
+# with glibc < 2.12
+LIBMDBX_CONF_OPTS = \
+ -DCMAKE_BUILD_TYPE=Release \
+ -DMDBX_INSTALL_MANPAGES=OFF \
+ -DBUILD_FOR_NATIVE_CPU=OFF \
+ -DMDBX_BUILD_CXX=$(if $(BR2_PACKAGE_LIBMDBX_CXX),ON,OFF) \
+ -DMDBX_BUILD_TOOLS=$(if $(BR2_PACKAGE_LIBMDBX_TOOLS),ON,OFF)
+
@@ -176,12 +150,16 @@ index 0000000000..f3720130ec
+endif
+
+ifeq ($(BR2_SHARED_LIBS)$(BR2_SHARED_STATIC_LIBS),y)
+LIBMDBX_CONF_OPTS += -DMDBX_BUILD_SHARED_LIBRARY=ON -DMDBX_LINK_TOOLS_NONSTATIC=ON
+LIBMDBX_CONF_OPTS += \
+ -DMDBX_BUILD_SHARED_LIBRARY=ON \
+ -DMDBX_LINK_TOOLS_NONSTATIC=ON
+else
+LIBMDBX_CONF_OPTS += -DMDBX_BUILD_SHARED_LIBRARY=OFF -DMDBX_LINK_TOOLS_NONSTATIC=OFF
+LIBMDBX_CONF_OPTS += \
+ -DMDBX_BUILD_SHARED_LIBRARY=OFF \
+ -DMDBX_LINK_TOOLS_NONSTATIC=OFF
+endif
+
+$(eval $(cmake-package))
--
2.29.2
2.33.0

View File

@@ -1,219 +0,0 @@
From 0bf9d06e8b090e2d9783d03074f3752ed708f6cf Mon Sep 17 00:00:00 2001
From: Leonid Yuriev <leo@yuriev.ru>
Date: Fri, 27 Nov 2020 16:31:12 +0300
Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Subject: [PATCH v5 0/1] cover letter for package/libmdbx: new package (library/database)
This patch adds libmdbx v0.9.2 and below is a brief overview of libmdbx.
Please merge.
Regards,
Leonid.
--
libmdbx is an extremely fast, compact, powerful, embedded, transactional
key-value database, with permissive license. libmdbx has a specific set
of properties and capabilities, focused on creating unique lightweight
solutions.
Historically, libmdbx (MDBX) is a deeply revised and extended descendant
of the legendary LMDB (Lightning Memory-Mapped Database). libmdbx
inherits all benefits from LMDB, but resolves some issues and adds a set
of improvements.
According to developers, for now libmdbx surpasses the LMDB in terms of
reliability, features and performance.
The most important differences MDBX from LMDB:
==============================================
1. More attention is paid to the quality of the code, to an
"unbreakability" of the API, to testing and automatic checks (i.e.
sanitizers, etc). So there:
- more control during operation;
- more checking parameters, internal audit of database structures;
- no warnings from compiler;
- no issues from ASAN, UBSAN, Valgrind, Coverity;
- etc.
2. Keys could be more than 2 times longer than LMDB.
3. Up to 20% faster than LMDB in CRUD benchmarks.
4. Automatic on-the-fly database size adjustment,
both increment and reduction.
5. Automatic continuous zero-overhead database compactification.
6. The same database format for 32- and 64-bit builds.
7. LIFO policy for Garbage Collection recycling (this can significantly
increase write performance due write-back disk cache up to several times
in a best case scenario).
8. Range query estimation.
9. Utility for checking the integrity of the database structure with
some recovery capabilities.
For more info please refer:
- https://github.com/erthink/libmdbx for source code and README.
- https://erthink.github.io/libmdbx for API description.
--
MDBX is a Btree-based database management library modeled loosely on the
BerkeleyDB API, but much simplified. The entire database (aka
"environment") is exposed in a memory map, and all data fetches return
data directly from the mapped memory, so no malloc's or memcpy's occur
during data fetches. As such, the library is extremely simple because it
requires no page caching layer of its own, and it is extremely high
performance and memory-efficient. It is also fully transactional with
full ACID semantics, and when the memory map is read-only, the database
integrity cannot be corrupted by stray pointer writes from application
code.
The library is fully thread-aware and supports concurrent read/write
access from multiple processes and threads. Data pages use a
copy-on-write strategy so no active data pages are ever overwritten,
which also provides resistance to corruption and eliminates the need of
any special recovery procedures after a system crash. Writes are fully
serialized; only one write transaction may be active at a time, which
guarantees that writers can never deadlock. The database structure is
multi-versioned so readers run with no locks; writers cannot block
readers, and readers don't block writers.
Unlike other well-known database mechanisms which use either write-ahead
transaction logs or append-only data writes, MDBX requires no
maintenance during operation. Both write-ahead loggers and append-only
databases require periodic checkpointing and/or compaction of their log
or database files otherwise they grow without bound. MDBX tracks
retired/freed pages within the database and re-uses them for new write
operations, so the database size does not grow without bound in normal
use.
The memory map can be used as a read-only or read-write map. It is
read-only by default as this provides total immunity to corruption.
Using read-write mode offers much higher write performance, but adds the
possibility for stray application writes thru pointers to silently
corrupt the database.
Features
========
- Key-value data model, keys are always sorted.
- Fully ACID-compliant, through to MVCC and CoW.
- Multiple key-value sub-databases within a single datafile.
- Range lookups, including range query estimation.
- Efficient support for short fixed length keys, including native
32/64-bit integers.
- Ultra-efficient support for multimaps. Multi-values sorted, searchable
and iterable. Keys stored without duplication.
- Data is memory-mapped and accessible directly/zero-copy. Traversal of
database records is extremely-fast.
- Transactions for readers and writers, ones do not block others.
- Writes are strongly serialized. No transaction conflicts nor
deadlocks.
- Readers are non-blocking, notwithstanding snapshot isolation.
- Nested write transactions.
- Reads scale linearly across CPUs.
- Continuous zero-overhead database compactification.
- Automatic on-the-fly database size adjustment.
- Customizable database page size.
- Olog(N) cost of lookup, insert, update, and delete operations by
virtue of B+ tree characteristics.
- Online hot backup.
- Append operation for efficient bulk insertion of pre-sorted data.
- No WAL nor any transaction journal. No crash recovery needed. No
maintenance is required.
- No internal cache and/or memory management, all done by basic OS
services.
Limitations
===========
- Page size: a power of 2, maximum 65536 bytes, default 4096 bytes.
- Key size: minimum 0, maximum ≈¼ pagesize (1300 bytes for default 4K
pagesize, 21780 bytes for 64K pagesize).
- Value size: minimum 0, maximum 2146435072 (0x7FF00000) bytes for maps,
≈¼ pagesize for multimaps (1348 bytes default 4K pagesize, 21828 bytes
for 64K pagesize).
- Write transaction size: up to 4194301 (0x3FFFFD) pages (16 GiB for
default 4K pagesize, 256 GiB for 64K pagesize).
- Database size: up to 2147483648 pages (8 TiB for default 4K pagesize,
128 TiB for 64K pagesize).
- Maximum sub-databases: 32765.
Gotchas
=======
- There cannot be more than one writer at a time, i.e. no more than one
write transaction at a time.
- libmdbx is based on B+ tree, so access to database pages is mostly
random. Thus SSDs provide a significant performance boost over
spinning disks for large databases.
- libmdbx uses shadow paging instead of WAL. Thus syncing data to disk
might be a bottleneck for write intensive workload.
- libmdbx uses copy-on-write for snapshot isolation during updates, but
read transactions prevents recycling an old retired/freed pages, since
it read ones. Thus altering of data during a parallel long-lived read
operation will increase the process work set, may exhaust entire free
database space, the database can grow quickly, and result in
performance degradation. Try to avoid long running read transactions.
- libmdbx is extraordinarily fast and provides minimal overhead for data
access, so you should reconsider using brute force techniques and
double check your code. On the one hand, in the case of libmdbx, a
simple linear search may be more profitable than complex indexes. On
the other hand, if you make something suboptimally, you can notice
detrimentally only on sufficiently large data.
--
Leonid Yuriev (1):
package/libmdbx: new package (library/database).
DEVELOPERS | 3 +++
package/Config.in | 1 +
package/libmdbx/Config.in | 45 ++++++++++++++++++++++++++++++++++++
package/libmdbx/libmdbx.hash | 5 ++++
package/libmdbx/libmdbx.mk | 33 ++++++++++++++++++++++++++
5 files changed, 87 insertions(+)
create mode 100644 package/libmdbx/Config.in
create mode 100644 package/libmdbx/libmdbx.hash
create mode 100644 package/libmdbx/libmdbx.mk
--
2.29.2

View File

@@ -4848,8 +4848,8 @@ status_done:
* могла быть выделена, а затем пролита в одной из родительских
* транзакций. Поэтому пока помещаем её в retired-список, который будет
* фильтроваться относительно dirty- и spilled-списков родительских
* транзакций при коммите
* дочерних транзакций, либо же будет записан в GC в неизменном виде. */
* транзакций при коммите дочерних транзакций, либо же будет записан
* в GC в неизменном виде. */
goto retire;
}
@@ -8682,7 +8682,7 @@ retry_noaccount:
mdbx_pnl_check4assert(txn->tw.reclaimed_pglist,
txn->mt_next_pgno - MDBX_ENABLE_REFUND));
mdbx_tassert(txn, mdbx_dirtylist_check(txn));
if (unlikely(/* paranoia */ loop > ((MDBX_DEBUG > 0) ? 9 : 99))) {
if (unlikely(/* paranoia */ loop > ((MDBX_DEBUG > 0) ? 12 : 42))) {
mdbx_error("too more loops %u, bailout", loop);
rc = MDBX_PROBLEM;
goto bailout;
@@ -8743,8 +8743,6 @@ retry_noaccount:
/* If using records from GC which we have not yet deleted,
* now delete them and any we reserved for tw.reclaimed_pglist. */
while (cleaned_gc_id <= txn->tw.last_reclaimed) {
gc_rid = cleaned_gc_id;
settled = 0;
rc = mdbx_cursor_first(&couple.outer, &key, NULL);
if (unlikely(rc != MDBX_SUCCESS)) {
if (rc == MDBX_NOTFOUND)
@@ -8756,6 +8754,9 @@ retry_noaccount:
rc = MDBX_CORRUPTED;
goto bailout;
}
gc_rid = cleaned_gc_id;
settled = 0;
reused_gc_slot = 0;
cleaned_gc_id = unaligned_peek_u64(4, key.iov_base);
if (!MDBX_DISABLE_PAGECHECKS &&
unlikely(cleaned_gc_id < MIN_TXNID || cleaned_gc_id > MAX_TXNID)) {
@@ -9121,6 +9122,7 @@ retry_noaccount:
} else if (rc != MDBX_NOTFOUND)
goto bailout;
txn->tw.last_reclaimed = gc_rid;
cleaned_gc_id = gc_rid + 1;
}
reservation_gc_id = gc_rid--;
mdbx_trace("%s: take @%" PRIaTXN " from head-gc-id", dbg_prefix_mode,
@@ -9198,7 +9200,7 @@ retry_noaccount:
key.iov_len = sizeof(reservation_gc_id);
key.iov_base = &reservation_gc_id;
data.iov_len = (chunk + 1) * sizeof(pgno_t);
mdbx_trace("%s.reserve: %u [%u...%u] @%" PRIaTXN, dbg_prefix_mode, chunk,
mdbx_trace("%s.reserve: %u [%u...%u) @%" PRIaTXN, dbg_prefix_mode, chunk,
settled + 1, settled + chunk + 1, reservation_gc_id);
mdbx_prep_backlog(txn, &couple.outer, data.iov_len);
rc = mdbx_cursor_put(&couple.outer, &key, &data,
@@ -9374,15 +9376,21 @@ retry_noaccount:
}
mdbx_tassert(txn, rc == MDBX_SUCCESS);
if (unlikely(txn->tw.loose_count != 0 ||
filled_gc_slot !=
(txn->tw.lifo_reclaimed
? (unsigned)MDBX_PNL_SIZE(txn->tw.lifo_reclaimed)
: 0))) {
mdbx_notice("** restart: reserve excess (filled-slot %u, loose-count %u)",
filled_gc_slot, txn->tw.loose_count);
if (unlikely(txn->tw.loose_count != 0)) {
mdbx_notice("** restart: got %u loose pages", txn->tw.loose_count);
goto retry;
}
if (unlikely(filled_gc_slot !=
(txn->tw.lifo_reclaimed
? (unsigned)MDBX_PNL_SIZE(txn->tw.lifo_reclaimed)
: 0))) {
const bool will_retry = loop < 9;
mdbx_notice("** %s: reserve excess (filled-slot %u, loop %u)",
will_retry ? "restart" : "ignore", filled_gc_slot, loop);
if (will_retry)
goto retry;
}
mdbx_tassert(txn,
txn->tw.lifo_reclaimed == NULL ||
@@ -10049,9 +10057,6 @@ int mdbx_txn_commit_ex(MDBX_txn *txn, MDBX_commit_latency *latency) {
ts_3 = latency ? mdbx_osal_monotime() : 0;
if (likely(rc == MDBX_SUCCESS)) {
if (txn->mt_dbs[MAIN_DBI].md_flags & DBI_DIRTY)
txn->mt_dbs[MAIN_DBI].md_mod_txnid = txn->mt_txnid;
txn->mt_dbs[FREE_DBI].md_mod_txnid = txn->mt_txnid;
MDBX_meta meta, *head = mdbx_meta_head(env);
memcpy(meta.mm_magic_and_version, head->mm_magic_and_version, 8);
@@ -10064,7 +10069,15 @@ int mdbx_txn_commit_ex(MDBX_txn *txn, MDBX_commit_latency *latency) {
meta.mm_geo = txn->mt_geo;
meta.mm_dbs[FREE_DBI] = txn->mt_dbs[FREE_DBI];
meta.mm_dbs[FREE_DBI].md_mod_txnid =
(txn->mt_dbistate[FREE_DBI] & DBI_DIRTY)
? txn->mt_txnid
: txn->mt_dbs[FREE_DBI].md_mod_txnid;
meta.mm_dbs[MAIN_DBI] = txn->mt_dbs[MAIN_DBI];
meta.mm_dbs[MAIN_DBI].md_mod_txnid =
(txn->mt_dbistate[MAIN_DBI] & DBI_DIRTY)
? txn->mt_txnid
: txn->mt_dbs[MAIN_DBI].md_mod_txnid;
meta.mm_canary = txn->mt_canary;
mdbx_meta_set_txnid(env, &meta, txn->mt_txnid);
@@ -10111,7 +10124,8 @@ static int mdbx_validate_meta(MDBX_env *env, MDBX_meta *const meta,
const uint64_t magic_and_version =
unaligned_peek_u64(4, &meta->mm_magic_and_version);
if (unlikely(magic_and_version != MDBX_DATA_MAGIC &&
magic_and_version != MDBX_DATA_MAGIC_DEVEL)) {
magic_and_version != MDBX_DATA_MAGIC_LEGACY_COMPAT &&
magic_and_version != MDBX_DATA_MAGIC_LEGACY_DEVEL)) {
mdbx_error("meta[%u] has invalid magic/version %" PRIx64, meta_number,
magic_and_version);
return ((magic_and_version >> 8) != MDBX_MAGIC) ? MDBX_INVALID
@@ -10423,10 +10437,9 @@ __cold static int mdbx_read_header(MDBX_env *env, MDBX_meta *dest,
}
if (dest->mm_psize == 0 ||
((env->me_stuck_meta < 0)
? (!META_IS_STEADY(dest) &&
!meta_weak_acceptable(env, dest, lck_exclusive))
: false)) {
(env->me_stuck_meta < 0 &&
!(META_IS_STEADY(dest) ||
meta_weak_acceptable(env, dest, lck_exclusive)))) {
mdbx_error("%s", "no usable meta-pages, database is corrupted");
if (rc == MDBX_SUCCESS) {
/* TODO: try to restore the database by fully checking b-tree structure
@@ -10475,8 +10488,8 @@ __cold static MDBX_page *mdbx_meta_model(const MDBX_env *env, MDBX_page *model,
pages2pv(pv2pages(model_meta->mm_geo.shrink_pv)));
model_meta->mm_psize = env->me_psize;
model_meta->mm_flags = (uint16_t)env->me_flags;
model_meta->mm_flags |=
model_meta->mm_flags = (uint16_t)env->me_flags & DB_PERSISTENT_FLAGS;
model_meta->mm_flags =
MDBX_INTEGERKEY; /* this is mm_dbs[FREE_DBI].md_flags */
model_meta->mm_dbs[FREE_DBI].md_root = P_INVALID;
model_meta->mm_dbs[MAIN_DBI].md_root = P_INVALID;
@@ -11050,11 +11063,12 @@ mdbx_env_set_geometry(MDBX_env *env, intptr_t size_lower, intptr_t size_now,
(env->me_txn0 && env->me_txn0->mt_owner == mdbx_thread_self());
#if MDBX_DEBUG
if (growth_step < 0)
if (growth_step < 0) {
growth_step = 1;
if (shrink_threshold < 0)
shrink_threshold = 1;
#endif
if (shrink_threshold < 0)
shrink_threshold = 1;
}
#endif /* MDBX_DEBUG */
intptr_t reasonable_maxsize = 0;
bool need_unlock = false;
@@ -11699,65 +11713,80 @@ __cold static int mdbx_setup_dxb(MDBX_env *env, const int lck_rc,
mdbx_assert(env, lck_rc == MDBX_RESULT_TRUE);
/* exclusive mode */
MDBX_meta clone;
MDBX_meta const *const steady = mdbx_meta_steady(env);
const txnid_t steady_txnid = mdbx_meta_txnid_fluid(env, steady);
MDBX_meta steady_clone;
err = mdbx_validate_meta_copy(env, steady, &steady_clone);
if (unlikely(err != MDBX_SUCCESS)) {
mdbx_error("meta[%u] with %s txnid %" PRIaTXN
" is corrupted, %s needed",
bytes2pgno(env, (uint8_t *)steady - env->me_map), "steady",
steady_txnid, "manual recovery");
return MDBX_CORRUPTED;
}
MDBX_meta const *const head = mdbx_meta_head(env);
if (steady == head)
break;
const txnid_t steady_txnid = mdbx_meta_txnid_fluid(env, steady);
if (META_IS_STEADY(steady)) {
err = mdbx_validate_meta_copy(env, steady, &clone);
if (unlikely(err != MDBX_SUCCESS)) {
mdbx_error("meta[%u] with %s txnid %" PRIaTXN
" is corrupted, %s needed",
bytes2pgno(env, (uint8_t *)steady - env->me_map), "steady",
steady_txnid, "manual recovery");
return MDBX_CORRUPTED;
}
if (steady == head)
break;
}
const pgno_t pgno = bytes2pgno(env, (uint8_t *)head - env->me_map);
const txnid_t head_txnid = mdbx_meta_txnid_fluid(env, head);
MDBX_meta head_clone;
const bool head_valid =
mdbx_validate_meta_copy(env, head, &head_clone) == MDBX_SUCCESS;
mdbx_validate_meta_copy(env, head, &clone) == MDBX_SUCCESS;
mdbx_assert(env, !META_IS_STEADY(steady) || head_txnid != steady_txnid);
if (unlikely(!head_valid)) {
mdbx_error("meta[%u] with %s txnid %" PRIaTXN
" is corrupted, %s needed",
pgno, "last", head_txnid, "rollback");
if (unlikely(!META_IS_STEADY(steady))) {
mdbx_error("%s for open or automatic rollback, %s",
"there are no suitable meta-pages",
"manual recovery is required");
return MDBX_CORRUPTED;
}
mdbx_warning("meta[%u] with last txnid %" PRIaTXN
" is corrupted, rollback needed",
pgno, head_txnid);
goto purge_meta_head;
}
mdbx_assert(env, head_txnid != head_txnid);
if (head_txnid == steady_txnid)
break;
mdbx_assert(env, META_IS_STEADY(steady) && !META_IS_STEADY(head));
if (meta_bootid_match(head)) {
mdbx_warning(
"opening after an unclean shutdown, but boot-id(%016" PRIx64
"-%016" PRIx64
") is MATCH: rollback NOT needed, steady-sync NEEDED%s",
bootid.x, bootid.y,
(env->me_flags & MDBX_RDONLY) ? ", but unable in read-only mode"
: "");
if (env->me_flags & MDBX_RDONLY)
if (env->me_flags & MDBX_RDONLY) {
mdbx_error("%s, but boot-id(%016" PRIx64 "-%016" PRIx64 ") is MATCH: "
"rollback NOT needed, steady-sync NEEDED%s",
"opening after an unclean shutdown", bootid.x, bootid.y,
", but unable in read-only mode");
return MDBX_WANNA_RECOVERY;
meta = head_clone;
}
mdbx_warning("%s, but boot-id(%016" PRIx64 "-%016" PRIx64 ") is MATCH: "
"rollback NOT needed, steady-sync NEEDED%s",
"opening after an unclean shutdown", bootid.x, bootid.y,
"");
meta = clone;
atomic_store32(&env->me_lck->mti_unsynced_pages, meta.mm_geo.next,
mo_Relaxed);
break;
}
if (unlikely(!META_IS_STEADY(steady))) {
mdbx_error("%s, but %s for automatic rollback: %s",
"opening after an unclean shutdown",
"there are no suitable meta-pages",
"manual recovery is required");
return MDBX_CORRUPTED;
}
if (env->me_flags & MDBX_RDONLY) {
mdbx_error("rollback needed: (from head %" PRIaTXN
" to steady %" PRIaTXN "), but unable in read-only mode",
head_txnid, steady_txnid);
mdbx_error("%s and rollback needed: (from head %" PRIaTXN
" to steady %" PRIaTXN ")%s",
"opening after an unclean shutdown", head_txnid,
steady_txnid, ", but unable in read-only mode");
return MDBX_WANNA_RECOVERY;
}
purge_meta_head:
mdbx_notice("rollback: purge%s meta[%u] with%s txnid %" PRIaTXN,
mdbx_notice("%s and doing automatic rollback: "
"purge%s meta[%u] with%s txnid %" PRIaTXN,
"opening after an unclean shutdown",
head_valid ? "" : " invalid", pgno, head_valid ? " weak" : "",
head_txnid);
mdbx_ensure(env, META_IS_STEADY(steady));
err = mdbx_override_meta(env, pgno, 0, head_valid ? head : steady);
if (err) {
mdbx_error("rollback: overwrite meta[%u] with txnid %" PRIaTXN
@@ -11837,59 +11866,78 @@ __cold static int mdbx_setup_dxb(MDBX_env *env, const int lck_rc,
}
}
//------------------------------------------------- setup madvise/readahead
atomic_store32(&env->me_lck->mti_discarded_tail,
bytes2pgno(env, used_aligned2os_bytes), mo_Relaxed);
#if MDBX_ENABLE_MADVISE
if (used_aligned2os_bytes < env->me_dxb_mmap.current) {
#if defined(MADV_REMOVE)
if ((env->me_flags & MDBX_WRITEMAP) != 0 &&
/* not recovery mode */ env->me_stuck_meta < 0) {
mdbx_notice("open-MADV_%s %u..%u", "REMOVE (deallocate file space)",
env->me_lck->mti_discarded_tail.weak,
bytes2pgno(env, env->me_dxb_mmap.current));
err = madvise(env->me_map + used_aligned2os_bytes,
env->me_dxb_mmap.current - used_aligned2os_bytes,
MADV_REMOVE)
? ignore_enosys(errno)
: MDBX_SUCCESS;
if (unlikely(MDBX_IS_ERROR(err)))
return err;
if ((env->me_flags & MDBX_RDONLY) == 0 && env->me_stuck_meta < 0) {
for (int n = 0; n < 3; ++n) {
MDBX_meta *const meta = METAPAGE(env, n);
if (unlikely(unaligned_peek_u64(4, &meta->mm_magic_and_version) !=
MDBX_DATA_MAGIC)) {
const txnid_t txnid = mdbx_meta_txnid_fluid(env, meta);
mdbx_notice("%s %s"
"meta[%u], txnid %" PRIaTXN,
"updating db-format signature for",
META_IS_STEADY(meta) ? "stead-" : "weak-", n, txnid);
err = mdbx_override_meta(env, n, txnid, meta);
if (unlikely(err != MDBX_SUCCESS)) {
mdbx_error("%s meta[%u], txnid %" PRIaTXN ", error %d",
"updating db-format signature for", n, txnid, err);
return err;
}
}
}
#endif /* MADV_REMOVE */
#if defined(MADV_DONTNEED)
mdbx_notice("open-MADV_%s %u..%u", "DONTNEED",
}
} /* lck exclusive, lck_rc == MDBX_RESULT_TRUE */
//---------------------------------------------------- setup madvise/readahead
#if MDBX_ENABLE_MADVISE
if (used_aligned2os_bytes < env->me_dxb_mmap.current) {
#if defined(MADV_REMOVE)
if (lck_rc && (env->me_flags & MDBX_WRITEMAP) != 0 &&
/* not recovery mode */ env->me_stuck_meta < 0) {
mdbx_notice("open-MADV_%s %u..%u", "REMOVE (deallocate file space)",
env->me_lck->mti_discarded_tail.weak,
bytes2pgno(env, env->me_dxb_mmap.current));
err = madvise(env->me_map + used_aligned2os_bytes,
env->me_dxb_mmap.current - used_aligned2os_bytes,
MADV_DONTNEED)
? ignore_enosys(errno)
: MDBX_SUCCESS;
err =
madvise(env->me_map + used_aligned2os_bytes,
env->me_dxb_mmap.current - used_aligned2os_bytes, MADV_REMOVE)
? ignore_enosys(errno)
: MDBX_SUCCESS;
if (unlikely(MDBX_IS_ERROR(err)))
return err;
#elif defined(POSIX_MADV_DONTNEED)
err = ignore_enosys(
posix_madvise(env->me_map + used_aligned2os_bytes,
env->me_dxb_mmap.current - used_aligned2os_bytes,
POSIX_MADV_DONTNEED));
if (unlikely(MDBX_IS_ERROR(err)))
return err;
#elif defined(POSIX_FADV_DONTNEED)
err = ignore_enosys(
posix_fadvise(env->me_lazy_fd, used_aligned2os_bytes,
env->me_dxb_mmap.current - used_aligned2os_bytes,
POSIX_FADV_DONTNEED));
if (unlikely(MDBX_IS_ERROR(err)))
return err;
#endif /* MADV_DONTNEED */
}
err = mdbx_set_readahead(env, bytes2pgno(env, used_bytes), readahead, true);
if (unlikely(err != MDBX_SUCCESS))
#endif /* MADV_REMOVE */
#if defined(MADV_DONTNEED)
mdbx_notice("open-MADV_%s %u..%u", "DONTNEED",
env->me_lck->mti_discarded_tail.weak,
bytes2pgno(env, env->me_dxb_mmap.current));
err =
madvise(env->me_map + used_aligned2os_bytes,
env->me_dxb_mmap.current - used_aligned2os_bytes, MADV_DONTNEED)
? ignore_enosys(errno)
: MDBX_SUCCESS;
if (unlikely(MDBX_IS_ERROR(err)))
return err;
#elif defined(POSIX_MADV_DONTNEED)
err = ignore_enosys(posix_madvise(
env->me_map + used_aligned2os_bytes,
env->me_dxb_mmap.current - used_aligned2os_bytes, POSIX_MADV_DONTNEED));
if (unlikely(MDBX_IS_ERROR(err)))
return err;
#elif defined(POSIX_FADV_DONTNEED)
err = ignore_enosys(posix_fadvise(
env->me_lazy_fd, used_aligned2os_bytes,
env->me_dxb_mmap.current - used_aligned2os_bytes, POSIX_FADV_DONTNEED));
if (unlikely(MDBX_IS_ERROR(err)))
return err;
#endif /* MADV_DONTNEED */
}
err = mdbx_set_readahead(env, bytes2pgno(env, used_bytes), readahead, true);
if (unlikely(err != MDBX_SUCCESS))
return err;
#endif /* MDBX_ENABLE_MADVISE */
} /* lck exclusive, lck_rc == MDBX_RESULT_TRUE */
return rc;
}
@@ -12227,7 +12275,7 @@ __cold int mdbx_env_turn_for_recovery(MDBX_env *env, unsigned target) {
} else {
txnid_t txnid = mdbx_meta_txnid_stable(env, &meta);
if (new_txnid <= txnid)
safe64_txnid_next(new_txnid);
new_txnid = safe64_txnid_next(txnid);
}
}
@@ -12908,25 +12956,26 @@ static int __hot cmp_int_unaligned(const MDBX_val *a, const MDBX_val *b) {
/* Compare two items lexically */
static int __hot cmp_lexical(const MDBX_val *a, const MDBX_val *b) {
if (a->iov_len == b->iov_len)
return memcmp(a->iov_base, b->iov_base, a->iov_len);
return a->iov_len ? memcmp(a->iov_base, b->iov_base, a->iov_len) : 0;
const int diff_len = (a->iov_len < b->iov_len) ? -1 : 1;
const size_t shortest = (a->iov_len < b->iov_len) ? a->iov_len : b->iov_len;
int diff_data = memcmp(a->iov_base, b->iov_base, shortest);
int diff_data = shortest ? memcmp(a->iov_base, b->iov_base, shortest) : 0;
return likely(diff_data) ? diff_data : diff_len;
}
/* Compare two items in reverse byte order */
static int __hot cmp_reverse(const MDBX_val *a, const MDBX_val *b) {
const uint8_t *pa = (const uint8_t *)a->iov_base + a->iov_len;
const uint8_t *pb = (const uint8_t *)b->iov_base + b->iov_len;
const size_t shortest = (a->iov_len < b->iov_len) ? a->iov_len : b->iov_len;
const uint8_t *const end = pa - shortest;
while (pa != end) {
int diff = *--pa - *--pb;
if (likely(diff))
return diff;
if (likely(shortest)) {
const uint8_t *pa = (const uint8_t *)a->iov_base + a->iov_len;
const uint8_t *pb = (const uint8_t *)b->iov_base + b->iov_len;
const uint8_t *const end = pa - shortest;
do {
int diff = *--pa - *--pb;
if (likely(diff))
return diff;
} while (pa != end);
}
return CMP2INT(a->iov_len, b->iov_len);
}
@@ -12934,7 +12983,9 @@ static int __hot cmp_reverse(const MDBX_val *a, const MDBX_val *b) {
/* Fast non-lexically comparator */
static int __hot cmp_lenfast(const MDBX_val *a, const MDBX_val *b) {
int diff = CMP2INT(a->iov_len, b->iov_len);
return likely(diff) ? diff : memcmp(a->iov_base, b->iov_base, a->iov_len);
return likely(diff || a->iov_len == 0)
? diff
: memcmp(a->iov_base, b->iov_base, a->iov_len);
}
static bool unsure_equal(MDBX_cmp_func cmp, const MDBX_val *a,
@@ -13086,10 +13137,11 @@ mdbx_page_get_ex(MDBX_cursor *const mc, const pgno_t pgno,
mdbx_tassert(txn, front <= txn->mt_front);
if (unlikely(pgno >= txn->mt_next_pgno)) {
mdbx_error("page #%" PRIaPGNO " beyond next-pgno", pgno);
notfound:
ret.page = nullptr;
corrupted:
mc->mc_txn->mt_flags |= MDBX_TXN_ERROR;
ret.err = MDBX_PAGE_NOTFOUND;
bailout:
mc->mc_txn->mt_flags |= MDBX_TXN_ERROR;
return ret;
}
@@ -13129,33 +13181,36 @@ dirty:
"mismatch actual pgno (%" PRIaPGNO ") != expected (%" PRIaPGNO
")\n",
ret.page->mp_pgno, pgno);
goto corrupted;
goto notfound;
}
#if !MDBX_DISABLE_PAGECHECKS
if (unlikely(ret.page->mp_flags & P_ILL_BITS)) {
bad_page(ret.page, "invalid page's flags (%u)\n", ret.page->mp_flags);
goto corrupted;
ret.err =
bad_page(ret.page, "invalid page's flags (%u)\n", ret.page->mp_flags);
goto bailout;
}
if (unlikely(ret.page->mp_txnid > front) &&
(ret.page->mp_txnid > txn->mt_front || front < txn->mt_txnid)) {
bad_page(ret.page,
"invalid page txnid (%" PRIaTXN ") for %s' txnid (%" PRIaTXN ")\n",
ret.page->mp_txnid,
(front == txn->mt_front && front != txn->mt_txnid) ? "front-txn"
: "parent-page",
front);
goto corrupted;
unlikely(ret.page->mp_txnid > txn->mt_front || front < txn->mt_txnid)) {
ret.err = bad_page(
ret.page,
"invalid page txnid (%" PRIaTXN ") for %s' txnid (%" PRIaTXN ")\n",
ret.page->mp_txnid,
(front == txn->mt_front && front != txn->mt_txnid) ? "front-txn"
: "parent-page",
front);
goto bailout;
}
if (unlikely((ret.page->mp_upper < ret.page->mp_lower ||
((ret.page->mp_lower | ret.page->mp_upper) & 1) ||
PAGEHDRSZ + ret.page->mp_upper > env->me_psize) &&
!IS_OVERFLOW(ret.page))) {
bad_page(ret.page, "invalid page lower(%u)/upper(%u) with limit (%u)\n",
ret.page->mp_lower, ret.page->mp_upper, page_space(env));
goto corrupted;
ret.err =
bad_page(ret.page, "invalid page lower(%u)/upper(%u) with limit (%u)\n",
ret.page->mp_lower, ret.page->mp_upper, page_space(env));
goto bailout;
}
#endif /* !MDBX_DISABLE_PAGECHECKS */
@@ -13393,12 +13448,14 @@ __hot static int mdbx_page_search(MDBX_cursor *mc, const MDBX_val *key,
: mc->mc_txn->mt_txnid;
MDBX_txn *scan = mc->mc_txn;
do
if (scan->mt_dbistate[mc->mc_dbi] & DBI_DIRTY) {
if ((scan->mt_flags & MDBX_TXN_DIRTY) &&
(mc->mc_dbi == MAIN_DBI ||
(scan->mt_dbistate[mc->mc_dbi] & DBI_DIRTY))) {
pp_txnid = scan->mt_front;
break;
}
while ((scan = scan->mt_parent) != nullptr);
if (unlikely((rc = mdbx_page_get(mc, root, &mc->mc_pg[0], pp_txnid) != 0)))
while (unlikely((scan = scan->mt_parent) != nullptr));
if (unlikely((rc = mdbx_page_get(mc, root, &mc->mc_pg[0], pp_txnid)) != 0))
return rc;
}
@@ -13524,8 +13581,10 @@ int mdbx_get_ex(MDBX_txn *txn, MDBX_dbi dbi, MDBX_val *key, MDBX_val *data,
MDBX_node *node = page_node(cx.outer.mc_pg[cx.outer.mc_top],
cx.outer.mc_ki[cx.outer.mc_top]);
if (F_ISSET(node_flags(node), F_DUPDATA)) {
// coverity[uninit_use : FALSE]
mdbx_tassert(txn, cx.outer.mc_xcursor == &cx.inner &&
(cx.inner.mx_cursor.mc_flags & C_INITIALIZED));
// coverity[uninit_use : FALSE]
*values_count =
(sizeof(*values_count) >= sizeof(cx.inner.mx_db.md_entries) ||
cx.inner.mx_db.md_entries <= PTRDIFF_MAX)
@@ -17273,18 +17332,20 @@ __cold static int mdbx_page_check(MDBX_cursor *const mc,
break;
case F_DUPDATA /* short sub-page */:
if (unlikely(dsize <= PAGEHDRSZ)) {
rc = bad_page(mp, "invalid nested-page record size (%zu)\n", dsize);
rc = bad_page(mp, "invalid nested/sub-page record size (%zu)\n",
dsize);
continue;
} else {
const MDBX_page *const sp = (MDBX_page *)data;
const char *const end_of_subpage = data + dsize;
const int nsubkeys = page_numkeys(sp);
switch (sp->mp_flags) {
switch (sp->mp_flags & /* ignore legacy P_DIRTY flag */ ~0x10) {
case P_LEAF | P_SUBP:
case P_LEAF | P_LEAF2 | P_SUBP:
break;
default:
rc = bad_page(mp, "invalid nested-page flags (%u)\n", sp->mp_flags);
rc = bad_page(mp, "invalid nested/sub-page flags (0x%02x)\n",
sp->mp_flags);
continue;
}
@@ -19234,7 +19295,7 @@ __cold int mdbx_dbi_dupsort_depthmask(MDBX_txn *txn, MDBX_dbi dbi,
break;
case F_DUPDATA | F_SUBDATA:
/* sub-tree */
*mask |= 1 << unaligned_peek_u16(1, &db->md_depth);
*mask |= 1 << UNALIGNED_PEEK_16(db, MDBX_db, md_depth);
break;
default:
mdbx_error("wrong node-flags %u", flags);
@@ -19680,7 +19741,7 @@ static int dbi_open(MDBX_txn *txn, const char *table_name, unsigned user_flags,
/* Got info, register DBI in this txn */
memset(txn->mt_dbxs + slot, 0, sizeof(MDBX_dbx));
txn->mt_dbs[slot] = *(MDBX_db *)data.iov_base;
memcpy(&txn->mt_dbs[slot], data.iov_base, sizeof(MDBX_db));
env->me_dbflags[slot] = 0;
rc = mdbx_dbi_bind(txn, slot, user_flags, keycmp, datacmp);
if (unlikely(rc != MDBX_SUCCESS)) {
@@ -20545,7 +20606,7 @@ __cold static int mdbx_walk_tree(mdbx_walk_ctx_t *ctx, const pgno_t pgno,
size_t subalign_bytes = 0;
MDBX_page_type_t subtype;
switch (sp->mp_flags) {
switch (sp->mp_flags & /* ignore legacy P_DIRTY flag */ ~0x10) {
case P_LEAF | P_SUBP:
subtype = MDBX_subpage_leaf;
break;

View File

@@ -67,7 +67,7 @@
#endif
#ifndef _CRT_SECURE_NO_WARNINGS
#define _CRT_SECURE_NO_WARNINGS
#endif
#endif /* _CRT_SECURE_NO_WARNINGS */
#if _MSC_VER > 1800
#pragma warning(disable : 4464) /* relative include path contains '..' */
#endif
@@ -384,7 +384,7 @@ MDBX_MAYBE_UNUSED static
#define MDBX_MAGIC UINT64_C(/* 56-bit prime */ 0x59659DBDEF4C11)
/* FROZEN: The version number for a database's datafile format. */
#define MDBX_DATA_VERSION 2
#define MDBX_DATA_VERSION 3
/* The version number for a database's lockfile format. */
#define MDBX_LOCK_VERSION 4
@@ -437,7 +437,7 @@ typedef uint16_t indx_t;
/*----------------------------------------------------------------------------*/
/* Core structures for database and shared memory (i.e. format definition) */
#pragma pack(push, 1)
#pragma pack(push, 4)
/* Information about a single database in the environment. */
typedef struct MDBX_db {
@@ -517,6 +517,8 @@ typedef struct MDBX_meta {
} MDBX_meta;
#pragma pack(1)
/* Common header for all page types. The page type depends on mp_flags.
*
* P_BRANCH and P_LEAF pages have unsorted 'MDBX_node's at the end, with
@@ -787,7 +789,11 @@ typedef struct MDBX_lockinfo {
#define MDBX_DATA_MAGIC \
((MDBX_MAGIC << 8) + MDBX_PNL_ASCENDING * 64 + MDBX_DATA_VERSION)
#define MDBX_DATA_MAGIC_DEVEL ((MDBX_MAGIC << 8) + 255)
#define MDBX_DATA_MAGIC_LEGACY_COMPAT \
((MDBX_MAGIC << 8) + MDBX_PNL_ASCENDING * 64 + 2)
#define MDBX_DATA_MAGIC_LEGACY_DEVEL ((MDBX_MAGIC << 8) + 255)
#define MDBX_LOCK_MAGIC ((MDBX_MAGIC << 8) + MDBX_LOCK_VERSION)

View File

@@ -5,9 +5,9 @@
// Non-inline part of the libmdbx C++ API (preliminary)
//
#ifdef _MSC_VER
#if defined(_MSC_VER) && !defined(_CRT_SECURE_NO_WARNINGS)
#define _CRT_SECURE_NO_WARNINGS
#endif
#endif /* _CRT_SECURE_NO_WARNINGS */
#include "../mdbx.h++"

View File

@@ -1781,6 +1781,7 @@ retry_mapview:;
if (limit < map->limit) {
/* unmap an excess at end of mapping. */
// coverity[offset_free : FALSE]
if (unlikely(munmap(map->dxb + limit, map->limit - limit)))
return errno;
map->limit = limit;
@@ -1854,6 +1855,7 @@ retry_mapview:;
if (unlikely(munmap(map->address, map->limit)))
return errno;
// coverity[pass_freed_arg : FALSE]
ptr = mmap(map->address, limit, mmap_prot,
(flags & MDBX_MRESIZE_MAY_MOVE)
? mmap_flags
@@ -1863,11 +1865,13 @@ retry_mapview:;
if (MAP_FIXED_NOREPLACE != 0 && MAP_FIXED_NOREPLACE != MAP_FIXED &&
unlikely(ptr == MAP_FAILED) && !(flags & MDBX_MRESIZE_MAY_MOVE) &&
errno == /* kernel don't support MAP_FIXED_NOREPLACE */ EINVAL)
// coverity[pass_freed_arg : FALSE]
ptr = mmap(map->address, limit, mmap_prot, mmap_flags | MAP_FIXED,
map->fd, 0);
if (unlikely(ptr == MAP_FAILED)) {
/* try to restore prev mapping */
// coverity[pass_freed_arg : FALSE]
ptr = mmap(map->address, map->limit, mmap_prot,
(flags & MDBX_MRESIZE_MAY_MOVE)
? mmap_flags
@@ -1877,6 +1881,7 @@ retry_mapview:;
if (MAP_FIXED_NOREPLACE != 0 && MAP_FIXED_NOREPLACE != MAP_FIXED &&
unlikely(ptr == MAP_FAILED) && !(flags & MDBX_MRESIZE_MAY_MOVE) &&
errno == /* kernel don't support MAP_FIXED_NOREPLACE */ EINVAL)
// coverity[pass_freed_arg : FALSE]
ptr = mmap(map->address, map->limit, mmap_prot, mmap_flags | MAP_FIXED,
map->fd, 0);
if (unlikely(ptr == MAP_FAILED)) {

View File

@@ -33,7 +33,7 @@
#if defined(_WIN32) || defined(_WIN64)
#if !defined(_CRT_SECURE_NO_WARNINGS)
#define _CRT_SECURE_NO_WARNINGS
#endif
#endif /* _CRT_SECURE_NO_WARNINGS */
#if !defined(_NO_CRT_STDIO_INLINE) && MDBX_BUILD_SHARED_LIBRARY && \
!defined(xMDBX_TOOLS) && MDBX_WITHOUT_MSVC_CRT
#define _NO_CRT_STDIO_INLINE

View File

@@ -23,7 +23,9 @@
#define _WIN32_WINNT 0x0601 /* Windows 7 */
#endif
#ifdef _MSC_VER
#ifndef _CRT_SECURE_NO_WARNINGS
#define _CRT_SECURE_NO_WARNINGS
#endif /* _CRT_SECURE_NO_WARNINGS */
#pragma warning(push, 1)
#pragma warning(disable : 4548) /* expression before comma has no effect; \
expected expression with side - effect */

View File

@@ -19,7 +19,7 @@
namespace chrono {
#pragma pack(push, 1)
#pragma pack(push, 4)
typedef union time {
uint64_t fixedpoint;

View File

@@ -135,8 +135,6 @@ inline bool parse_option_intptr(int argc, char *const argv[], int &narg,
//-----------------------------------------------------------------------------
#pragma pack(push, 1)
struct keygen_params_pod {
/* Параметры генератора пар key-value. Также может быть полезным описание
* алгоритма генерации в keygen.h
@@ -307,8 +305,6 @@ struct actor_config_pod {
wait4id(wait4id) {}
};
#pragma pack(pop)
extern const struct option_verb mode_bits[];
extern const struct option_verb table_bits[];
void dump(const char *title = "config-dump: ");

View File

@@ -609,10 +609,6 @@ int main(int argc, char *const argv[]) {
if (global::config::cleanup_before)
cleanup();
log_trace(">> probe entropy_ticks()");
entropy_ticks();
log_trace("<< probe entropy_ticks()");
if (global::actors.size() == 1) {
logging::setup("main");
global::singlemode = true;

View File

@@ -128,6 +128,8 @@ void testcase_nested::push_txn() {
std::move(speculum_snapshot));
log_verbose("begin level#%zu txn #%" PRIu64 ", flags 0x%x, serial %" PRIu64,
stack.size(), mdbx_txn_id(nested_txn), flags, serial);
if (!dbi && stack.size() == 1)
dbi = db_table_open(true);
}
bool testcase_nested::pop_txn(bool abort) {
@@ -139,6 +141,9 @@ bool testcase_nested::pop_txn(bool abort) {
log_verbose(
"abort level#%zu txn #%" PRIu64 ", undo serial %" PRIu64 " <- %" PRIu64,
stack.size(), mdbx_txn_id(txn), serial, std::get<1>(stack.top()));
if (dbi > 0 && stack.size() == 1 &&
is_handle_created_in_current_txn(dbi, txn))
dbi = 0;
int err = mdbx_txn_abort(txn);
if (unlikely(err != MDBX_SUCCESS))
failure_perror("mdbx_txn_abort()", err);
@@ -200,7 +205,7 @@ bool testcase_nested::trim_tail(unsigned window_width) {
while (fifo.size() > window_width) {
uint64_t tail_serial = fifo.back().first;
const unsigned tail_count = fifo.back().second;
log_verbose("nested: pop-tail (serial %" PRIu64 ", count %u)",
log_verbose("nested: trim-tail (serial %" PRIu64 ", count %u)",
tail_serial, tail_count);
fifo.pop_back();
for (unsigned n = 0; n < tail_count; ++n) {

365
test/stochastic_small.sh Executable file
View File

@@ -0,0 +1,365 @@
#!/usr/bin/env bash
if ! which make cc c++ tee >/dev/null; then
echo "Please install the following prerequisites: make cc c++ tee banner" >&2
exit 1
fi
LIST=--hill
FROM=1
UPTO=9999
MONITOR=
LOOPS=
SKIP_MAKE=no
BANNER="$(which banner 2>/dev/null | echo echo)"
UNAME="$(uname -s 2>/dev/null || echo Unknown)"
DB_UPTO_MB=17408
while [ -n "$1" ]
do
case "$1" in
--help)
echo "--multi Engage multi-process test scenario (default)"
echo "--single Execute series of single-process tests (for QEMU, etc)"
echo "--with-valgrind Run tests under Valgrind's memcheck tool"
echo "--skip-make Don't (re)build libmdbx and test's executable"
echo "--from NN Start iterating from the NN ops per test case"
echo "--upto NN Don't run tests with more than NN ops per test case"
echo "--loops NN Stop after the NN loops"
echo "--dir PATH Specifies directory for test DB and other files (it will be cleared)"
echo "--db-upto-mb NN Limits upper size of test DB to the NN megabytes"
echo "--help Print this usage help and exit"
exit -2
;;
--multi)
LIST=basic
;;
--single)
LIST="--nested --hill --append --ttl --copy"
;;
--with-valgrind)
echo " NOTE: Valgrind could produce some false-positive warnings"
echo " in multi-process environment with shared memory."
echo " For instance, when the process 'A' explicitly marks a memory"
echo " region as 'undefined', the process 'B' fill it,"
echo " and after this process 'A' read such region, etc."
MONITOR="valgrind --trace-children=yes --log-file=valgrind-%p.log --leak-check=full --track-origins=yes --error-exitcode=42 --suppressions=test/valgrind_suppress.txt"
rm -f valgrind-*.log
;;
--skip-make)
SKIP_MAKE=yes
;;
--from)
FROM=$(($2))
if [ -z "$FROM" -o "$FROM" -lt 1 ]; then
echo "Invalid value '$FROM' for --from option"
exit -2
fi
shift
;;
--upto)
UPTO=$(($2))
if [ -z "$UPTO" -o "$UPTO" -lt 1 ]; then
echo "Invalid value '$UPTO' for --upto option"
exit -2
fi
shift
;;
--loops)
LOOPS=$(($2))
if [ -z "$LOOPS" -o "$LOOPS" -lt 1 -o "$LOOPS" -gt 99 ]; then
echo "Invalid value '$LOOPS' for --loops option"
exit -2
fi
shift
;;
--dir)
TESTDB_DIR="$2"
if [ -z "$TESTDB_DIR" ]; then
echo "Invalid value '$TESTDB_DIR' for --dir option"
exit -2
fi
shift
;;
--db-upto-mb)
DB_UPTO_MB=$(($2))
if [ -z "$DB_UPTO_MB" -o "$DB_UPTO_MB" -lt 1 -o "$DB_UPTO_MB" -gt 4194304 ]; then
echo "Invalid value '$DB_UPTO_MB' for --db-upto-mb option"
exit -2
fi
shift
;;
*)
echo "Unknown option '$1'"
exit -2
;;
esac
shift
done
set -euo pipefail
if [ -z "$MONITOR" ]; then
if which time >/dev/null 2>/dev/null; then
MONITOR=$(which time)
if $MONITOR -o /dev/stdout true >/dev/null 2>/dev/null; then
MONITOR="$MONITOR -o /dev/stdout"
fi
fi
export MALLOC_CHECK_=7 MALLOC_PERTURB_=42
fi
###############################################################################
# 1. clean data from prev runs and examine available RAM
WANNA_MOUNT=0
case ${UNAME} in
Linux)
MAKE=make
if [ -z "${TESTDB_DIR:-}" ]; then
for old_test_dir in $(ls -d /dev/shm/mdbx-test.[0-9]* 2>/dev/null); do
rm -rf $old_test_dir
done
TESTDB_DIR="/dev/shm/mdbx-test.$$"
fi
mkdir -p $TESTDB_DIR && rm -f $TESTDB_DIR/*
if LC_ALL=C free | grep -q -i available; then
ram_avail_mb=$(($(LC_ALL=C free | grep -i Mem: | tr -s [:blank:] ' ' | cut -d ' ' -f 7) / 1024))
else
ram_avail_mb=$(($(LC_ALL=C free | grep -i Mem: | tr -s [:blank:] ' ' | cut -d ' ' -f 4) / 1024))
fi
;;
FreeBSD)
MAKE=gmake
if [ -z "${TESTDB_DIR:-}" ]; then
for old_test_dir in $(ls -d /tmp/mdbx-test.[0-9]* 2>/dev/null); do
umount $old_test_dir && rm -r $old_test_dir
done
TESTDB_DIR="/tmp/mdbx-test.$$"
rm -rf $TESTDB_DIR && mkdir -p $TESTDB_DIR
WANNA_MOUNT=1
else
mkdir -p $TESTDB_DIR && rm -f $TESTDB_DIR/*
fi
ram_avail_mb=$(($(LC_ALL=C vmstat -s | grep -ie '[0-9] pages free$' | cut -d p -f 1) * ($(LC_ALL=C vmstat -s | grep -ie '[0-9] bytes per page$' | cut -d b -f 1) / 1024) / 1024))
;;
Darwin)
MAKE=make
if [ -z "${TESTDB_DIR:-}" ]; then
for vol in $(ls -d /Volumes/mdx[0-9]*[0-9]tst 2>/dev/null); do
disk=$(mount | grep $vol | cut -d ' ' -f 1)
echo "umount: volume $vol disk $disk"
hdiutil unmount $vol -force
hdiutil detach $disk
done
TESTDB_DIR="/Volumes/mdx$$tst"
WANNA_MOUNT=1
else
mkdir -p $TESTDB_DIR && rm -f $TESTDB_DIR/*
fi
pagesize=$(($(LC_ALL=C vm_stat | grep -o 'page size of [0-9]\+ bytes' | cut -d' ' -f 4) / 1024))
freepages=$(LC_ALL=C vm_stat | grep '^Pages free:' | grep -o '[0-9]\+\.$' | cut -d'.' -f 1)
ram_avail_mb=$((pagesize * freepages / 1024))
echo "pagesize ${pagesize}K, freepages ${freepages}, ram_avail_mb ${ram_avail_mb}"
;;
*)
echo "FIXME: ${UNAME} not supported by this script"
exit 2
;;
esac
rm -f ${TESTDB_DIR}/*
###############################################################################
# 2. estimate reasonable RAM space for test-db
echo "=== ${ram_avail_mb}M RAM available"
ram_reserve4logs_mb=1234
if [ $ram_avail_mb -lt $ram_reserve4logs_mb ]; then
echo "=== At least ${ram_reserve4logs_mb}Mb RAM required"
exit 3
fi
#
# В режимах отличных от MDBX_WRITEMAP изменения до записи в файл
# будут накапливаться в памяти, что может потребовать свободной
# памяти размером с БД. Кроме этого, в тест входит сценарий
# создания копия БД на ходу. Поэтому БД не может быть больше 1/3
# от доступной памяти. Однако, следует учесть что malloc() будет
# не сразу возвращать выделенную память системе, а также
# предусмотреть места для логов.
#
# In non-MDBX_WRITEMAP modes, updates (dirty pages) will
# accumulate in memory before writing to the disk, which may
# require a free memory up to the size of a whole database. In
# addition, the test includes a script create a copy of the
# database on the go. Therefore, the database cannot be more 1/3
# of available memory. Moreover, should be taken into account
# that malloc() will not return the allocated memory to the
# system immediately, as well some space is required for logs.
#
db_size_mb=$(((ram_avail_mb - ram_reserve4logs_mb) / 4))
if [ $db_size_mb -gt $DB_UPTO_MB ]; then
db_size_mb=$DB_UPTO_MB
fi
echo "=== use ${db_size_mb}M for DB"
###############################################################################
# 3. Create test-directory in ramfs/tmpfs, i.e. create/format/mount if required
case ${UNAME} in
Linux)
;;
FreeBSD)
if [[ WANNA_MOUNT ]]; then
mount -t tmpfs tmpfs $TESTDB_DIR
fi
;;
Darwin)
if [[ WANNA_MOUNT ]]; then
ramdisk_size_mb=$((42 + db_size_mb * 2 + ram_reserve4logs_mb))
number_of_sectors=$((ramdisk_size_mb * 2048))
ramdev=$(hdiutil attach -nomount ram://${number_of_sectors})
diskutil erasevolume ExFAT "mdx$$tst" ${ramdev}
fi
;;
*)
echo "FIXME: ${UNAME} not supported by this script"
exit 2
;;
esac
###############################################################################
# 4. build the test executables
if [ "$SKIP_MAKE" != "yes" ]; then
${MAKE} -j$(which nproc >/dev/null 2>/dev/null && nproc || echo 2) build-test
fi
###############################################################################
# 5. run stochastic iterations
if which lz4 >/dev/null; then
function logger {
lz4 > ${TESTDB_DIR}/long.log.lz4
}
elif which gzip >/dev/null; then
function logger {
gzip > ${TESTDB_DIR}/long.log.gz
}
else
function logger {
cat > ${TESTDB_DIR}/long.log
}
fi
syncmodes=("" ,+nosync-safe ,+nosync-utterly)
options=(writemap coalesce lifo notls perturb)
function join { local IFS="$1"; shift; echo "$*"; }
function bits2options {
local bits=$1
local i
local list=()
for ((i = 0; i < ${#options[@]}; ++i)); do
list[$i]=$( (( (bits & (1 << i)) != 0 )) && echo -n '+' || echo -n '-'; echo ${options[$i]})
done
join , ${list[@]}
}
function failed {
echo "FAILED" >&2
exit 1
}
function check_deep {
if [ "$case" = "basic" -o "$case" = "--hill" ]; then
tee >(logger) | grep -e reach -e achieve
else
logger
fi
}
function probe {
echo "----------------------------------------------- $(date)"
echo "${caption}"
rm -f ${TESTDB_DIR}/* || failed
for case in $LIST
do
echo "Run ./mdbx_test ${speculum} --random-writemap=no --ignore-dbfull --repeat=1 --pathname=${TESTDB_DIR}/long.db --cleanup-after=no $@ $case"
${MONITOR} ./mdbx_test ${speculum} --random-writemap=no --ignore-dbfull --repeat=1 --pathname=${TESTDB_DIR}/long.db --cleanup-before=yes --cleanup-after=no "$@" $case | check_deep \
&& ${MONITOR} ./mdbx_chk ${TESTDB_DIR}/long.db | tee ${TESTDB_DIR}/long-chk.log \
&& ([ ! -e ${TESTDB_DIR}/long.db-copy ] || ${MONITOR} ./mdbx_chk ${TESTDB_DIR}/long.db-copy | tee ${TESTDB_DIR}/long-chk-copy.log) \
|| failed
done
}
#------------------------------------------------------------------------------
count=0
loop=0
cases='?'
for ((wbatch=FROM; wbatch<=UPTO; ++wbatch)); do
if [ -n "$LOOPS" ] && [ $loop -ge "$LOOPS" ]; then echo "The '--loops $LOOPS' limit reached"; break; fi
echo "======================================================================="
speculum=$([ $wbatch -le 1000 ] && echo '--speculum' || true)
nops=$((wbatch/7 + 1))
for ((rep=1; rep < 11; ++rep)); do
echo "======================================================================="
${BANNER} "$nops / $wbatch, repeat $rep"
subcase=0
for ((bits=2**${#options[@]}; --bits >= 0; )); do
seed=$(($(date +%s) + RANDOM))
split=30
caption="Probe #$((++count)) int-key,int-data, split=${split}, case $((++subcase)) of ${cases}" probe \
--pagesize=4K --size-upper-upto=${db_size_mb}M --table=+key.integer,+data.integer --keygen.split=${split} --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$nops --batch.write=$wbatch --mode=$(bits2options $bits)${syncmodes[count%3]} \
--keygen.seed=${seed}
split=24
caption="Probe #$((++count)) int-key,int-data, split=${split}, case $((++subcase)) of ${cases}" probe \
--pagesize=4K --size-upper-upto=${db_size_mb}M --table=+key.integer,+data.integer --keygen.split=${split} --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$nops --batch.write=$wbatch --mode=$(bits2options $bits)${syncmodes[count%3]} \
--keygen.seed=${seed}
split=16
caption="Probe #$((++count)) int-key,w/o-dups, split=${split}, case $((++subcase)) of ${cases}" probe \
--pagesize=4K --size-upper-upto=${db_size_mb}M --table=+key.integer,-data.dups --keygen.split=${split} --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$nops --batch.write=$wbatch --mode=$(bits2options $bits)${syncmodes[count%3]} \
--keygen.seed=${seed}
caption="Probe #$((++count)) int-key,int-data, split=${split}, case $((++subcase)) of ${cases}" probe \
--pagesize=4K --size-upper-upto=${db_size_mb}M --table=+key.integer,+data.integer --keygen.split=${split} --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$nops --batch.write=$wbatch --mode=$(bits2options $bits)${syncmodes[count%3]} \
--keygen.seed=${seed}
caption="Probe #$((++count)) w/o-dups, split=${split}, case $((++subcase)) of ${cases}" probe \
--pagesize=4K --size-upper-upto=${db_size_mb}M --table=-data.dups --keygen.split=${split} --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$nops --batch.write=$wbatch --mode=$(bits2options $bits)${syncmodes[count%3]} \
--keygen.seed=${seed}
split=4
caption="Probe #$((++count)) int-key,w/o-dups, split=${split}, case $((++subcase)) of ${cases}" probe \
--pagesize=4K --size-upper-upto=${db_size_mb}M --table=+key.integer,-data.dups --keygen.split=${split} --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$nops --batch.write=$wbatch --mode=$(bits2options $bits)${syncmodes[count%3]} \
--keygen.seed=${seed}
caption="Probe #$((++count)) int-key,int-data, split=${split}, case $((++subcase)) of ${cases}" probe \
--pagesize=4K --size-upper-upto=${db_size_mb}M --table=+key.integer,+data.integer --keygen.split=${split} --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$nops --batch.write=$wbatch --mode=$(bits2options $bits)${syncmodes[count%3]} \
--keygen.seed=${seed}
caption="Probe #$((++count)) w/o-dups, split=${split}, case $((++subcase)) of ${cases}" probe \
--pagesize=4K --size-upper-upto=${db_size_mb}M --table=-data.dups --keygen.split=${split} --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$nops --batch.write=$wbatch --mode=$(bits2options $bits)${syncmodes[count%3]} \
--keygen.seed=${seed}
done # options
cases="${subcase}"
done # repeats
loop=$((loop + 1))
if [ -n "$LOOPS" ] && [ $loop -ge "$LOOPS" ]; then break; fi
done # wbatch
echo "=== ALL DONE ====================== $(date)"

View File

@@ -477,6 +477,15 @@ void testcase::update_canary(uint64_t increment) {
log_trace("<< update_canary: sequence = %" PRIu64, canary_now.y);
}
bool testcase::is_handle_created_in_current_txn(const MDBX_dbi handle,
MDBX_txn *txn) {
unsigned flags, state;
int err = mdbx_dbi_flags_ex(txn, handle, &flags, &state);
if (unlikely(err != MDBX_SUCCESS))
failure_perror("mdbx_dbi_flags_ex()", err);
return (state & MDBX_DBI_CREAT) != 0;
}
int testcase::db_open__begin__table_create_open_clean(MDBX_dbi &handle) {
db_open();
@@ -484,6 +493,9 @@ int testcase::db_open__begin__table_create_open_clean(MDBX_dbi &handle) {
for (;;) {
txn_begin(false);
handle = db_table_open(true);
if (is_handle_created_in_current_txn(handle, txn_guard.get()))
return MDBX_SUCCESS;
db_table_clear(handle);
err = breakable_commit();
if (likely(err == MDBX_SUCCESS)) {

View File

@@ -268,6 +268,7 @@ protected:
void db_table_clear(MDBX_dbi handle, MDBX_txn *txn = nullptr);
void db_table_close(MDBX_dbi handle);
int db_open__begin__table_create_open_clean(MDBX_dbi &handle);
bool is_handle_created_in_current_txn(const MDBX_dbi handle, MDBX_txn *txn);
bool wait4start();
void report(size_t nops_done);

View File

@@ -101,164 +101,6 @@ bool is_samedata(const MDBX_val *a, const MDBX_val *b) {
//-----------------------------------------------------------------------------
/* TODO: replace my 'libmera' from t1ha. */
uint64_t entropy_ticks(void) {
#if defined(EMSCRIPTEN)
return (uint64_t)emscripten_get_now();
#endif /* EMSCRIPTEN */
#if defined(__APPLE__) || defined(__MACH__)
return mach_absolute_time();
#endif /* defined(__APPLE__) || defined(__MACH__) */
#if defined(__sun__) || defined(__sun)
return gethrtime();
#endif /* __sun__ */
#if defined(__GNUC__) || defined(__clang__)
#if defined(__ia64__)
uint64_t ticks;
__asm __volatile("mov %0=ar.itc" : "=r"(ticks));
return ticks;
#elif defined(__hppa__)
uint64_t ticks;
__asm __volatile("mfctl 16, %0" : "=r"(ticks));
return ticks;
#elif defined(__s390__)
uint64_t ticks;
__asm __volatile("stck 0(%0)" : : "a"(&(ticks)) : "memory", "cc");
return ticks;
#elif defined(__alpha__) || defined(__alpha)
uint64_t ticks;
__asm __volatile("rpcc %0" : "=r"(ticks));
return ticks;
#elif defined(__sparc__) || defined(__sparc) || defined(__sparc64__) || \
defined(__sparc64) || defined(__sparc_v8plus__) || \
defined(__sparc_v8plus) || defined(__sparc_v8plusa__) || \
defined(__sparc_v8plusa) || defined(__sparc_v9__) || defined(__sparc_v9)
union {
uint64_t u64;
struct {
#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
uint32_t h, l;
#else
uint32_t l, h;
#endif
} u32;
} cycles;
#if defined(__sparc_v8plus__) || defined(__sparc_v8plusa__) || \
defined(__sparc_v9__) || defined(__sparc_v8plus) || \
defined(__sparc_v8plusa) || defined(__sparc_v9)
#if UINTPTR_MAX > 0xffffFFFFul || ULONG_MAX > 0xffffFFFFul || \
defined(__sparc64__) || defined(__sparc64)
__asm __volatile("rd %%tick, %0" : "=r"(cycles.u64));
#else
__asm __volatile("rd %%tick, %1; srlx %1, 32, %0"
: "=r"(cycles.u32.h), "=r"(cycles.u32.l));
#endif /* __sparc64__ */
#else
__asm __volatile(".byte 0x83, 0x41, 0x00, 0x00; mov %%g1, %0"
: "=r"(cycles.u64)
:
: "%g1");
#endif /* __sparc8plus__ || __sparc_v9__ */
return cycles.u64;
#elif (defined(__powerpc64__) || defined(__ppc64__) || defined(__ppc64) || \
defined(__powerpc64))
uint64_t ticks;
__asm __volatile("mfspr %0, 268" : "=r"(ticks));
return ticks;
#elif (defined(__powerpc__) || defined(__ppc__) || defined(__powerpc) || \
defined(__ppc))
#if UINTPTR_MAX > 0xffffFFFFul || ULONG_MAX > 0xffffFFFFul
uint64_t ticks;
__asm __volatile("mftb %0" : "=r"(ticks));
*now = ticks;
#else
uint64_t ticks;
uint32_t low, high_before, high_after;
__asm __volatile("mftbu %0; mftb %1; mftbu %2"
: "=r"(high_before), "=r"(low), "=r"(high_after));
ticks = (uint64_t)high_after << 32;
ticks |= low & /* zeroes if high part has changed */
~(high_before - high_after);
#endif
#elif (defined(__aarch64__) || (defined(__ARM_ARCH) && __ARM_ARCH > 7)) && \
!defined(MDBX_SAFE4QEMU)
uint64_t virtual_timer;
__asm __volatile("mrs %0, cntvct_el0" : "=r"(virtual_timer));
return virtual_timer;
#elif (defined(__ARM_ARCH) && __ARM_ARCH > 5 && __ARM_ARCH < 8) || \
defined(_M_ARM)
static uint32_t pmcntenset = 0x00425B00;
if (unlikely(pmcntenset == 0x00425B00)) {
uint32_t pmuseren;
#ifdef _M_ARM
pmuseren = _MoveFromCoprocessor(15, 0, 9, 14, 0);
#else
__asm("mrc p15, 0, %0, c9, c14, 0" : "=r"(pmuseren));
#endif
if (1 & pmuseren /* Is it allowed for user mode code? */) {
#ifdef _M_ARM
pmcntenset = _MoveFromCoprocessor(15, 0, 9, 12, 1);
#else
__asm("mrc p15, 0, %0, c9, c12, 1" : "=r"(pmcntenset));
#endif
} else
pmcntenset = 0;
}
if (pmcntenset & 0x80000000ul /* Is it counting? */) {
#ifdef _M_ARM
return __rdpmccntr64();
#else
uint32_t pmccntr;
__asm __volatile("mrc p15, 0, %0, c9, c13, 0" : "=r"(pmccntr));
return pmccntr;
#endif
}
#elif ((defined(_MIPS_ISA) && defined(_MIPS_ISA_MIPS2) && \
_MIPS_ISA >= _MIPS_ISA_MIPS2) || \
(defined(__mips) && __mips >= 2) || defined(_R4000)) && \
!defined(MDBX_SAFE4QEMU) /* QEMU may not emulate the CC register \
(High-resolution cycle counter) */
unsigned count;
__asm __volatile("rdhwr %0, $2" : "=r"(count));
return count;
#endif /* arch selector */
#endif /* __GNUC__ || __clang__ */
#if defined(__e2k__) || defined(__ia32__)
return __rdtsc();
#elif defined(_WIN32) || defined(_WIN64) || defined(_WINDOWS)
LARGE_INTEGER PerformanceCount;
if (QueryPerformanceCounter(&PerformanceCount))
return PerformanceCount.QuadPart;
return GetTickCount64();
#else
struct timespec ts;
#if defined(CLOCK_MONOTONIC_COARSE)
clockid_t clk_id = CLOCK_MONOTONIC_COARSE;
#elif defined(CLOCK_MONOTONIC_RAW)
clockid_t clk_id = CLOCK_MONOTONIC_RAW;
#else
clockid_t clk_id = CLOCK_MONOTONIC;
#endif
int rc = clock_gettime(clk_id, &ts);
if (unlikely(rc))
failure_perror("clock_gettime()", rc);
return (((uint64_t)ts.tv_sec) << 32) + ts.tv_nsec;
#endif
}
//-----------------------------------------------------------------------------
uint64_t prng64_white(uint64_t &state) {
state = prng64_map2_careless(state);
return bleach64(state);
@@ -303,8 +145,6 @@ uint64_t prng64(void) { return prng64_white(prng_state); }
void prng_fill(void *ptr, size_t bytes) { prng_fill(prng_state, ptr, bytes); }
uint64_t entropy_white() { return bleach64(entropy_ticks()); }
double double_from_lower(uint64_t salt) {
#ifdef IEEE754_DOUBLE_BIAS
ieee754_double r;
@@ -336,25 +176,25 @@ double double_from_upper(uint64_t salt) {
#endif
}
bool flipcoin() { return bleach32((uint32_t)entropy_ticks()) & 1; }
bool flipcoin_x2() { return (bleach32((uint32_t)entropy_ticks()) & 3) == 0; }
bool flipcoin_x3() { return (bleach32((uint32_t)entropy_ticks()) & 7) == 0; }
bool flipcoin_x4() { return (bleach32((uint32_t)entropy_ticks()) & 15) == 0; }
bool flipcoin() { return prng32() & 1; }
bool flipcoin_x2() { return (prng32() & 3) == 0; }
bool flipcoin_x3() { return (prng32() & 7) == 0; }
bool flipcoin_x4() { return (prng32() & 15) == 0; }
bool flipcoin_n(unsigned n) {
return (bleach64(entropy_ticks()) & ((UINT64_C(1) << n) - 1)) == 0;
return (prng64() & ((UINT64_C(1) << n) - 1)) == 0;
}
bool jitter(unsigned probability_percent) {
const uint32_t top = UINT32_MAX - UINT32_MAX % 100;
uint32_t dice, edge = (top) / 100 * probability_percent;
do
dice = bleach32((uint32_t)entropy_ticks());
dice = prng32();
while (dice >= top);
return dice < edge;
}
void jitter_delay(bool extra) {
unsigned dice = entropy_white() & 3;
unsigned dice = prng32() & 3;
if (dice == 0) {
log_trace("== jitter.no-delay");
} else {
@@ -367,8 +207,8 @@ void jitter_delay(bool extra) {
osal_yield();
cpu_relax();
if (dice > 2) {
unsigned us = entropy_white() &
(extra ? 0xffff /* 656 ms */ : 0x3ff /* 1 ms */);
unsigned us =
prng32() & (extra ? 0xffff /* 656 ms */ : 0x3ff /* 1 ms */);
log_trace("== jitter.delay: %0.6f", us / 1000000.0);
osal_udelay(us);
}

View File

@@ -292,8 +292,6 @@ inline bool is_samedata(const MDBX_val &a, const MDBX_val &b) {
}
std::string format(const char *fmt, ...);
uint64_t entropy_ticks(void);
uint64_t entropy_white(void);
static inline uint64_t bleach64(uint64_t v) {
// Tommy Ettinger, https://www.blogger.com/profile/04953541827437796598
// http://mostlymangling.blogspot.com/2019/01/better-stronger-mixer-and-test-procedure.html