Compare commits

...

91 Commits

Author SHA1 Message Date
moneromooo-monero
a3aa2b5a57 mdbx-doc: import - mdb_cursor_del does not invalidate the cursor (ITS#8857). 2018-09-24 19:35:39 +03:00
Howard Chu
02276500c9 mdbx-doc: import - GET_MULTIPLE etc don't return the key (ITS#8908).
Unnecessary since these are DUPs, the key will always be the same
2018-09-24 19:35:36 +03:00
Leonid Yuriev
de44ecccd1 mdbx: backport - update MAX_PAGENO and MAX_MAPSIZE64. 2018-09-23 18:07:29 +03:00
Leonid Yuriev
5049c86517 mdbx: backport - avoid empty and unneeded large/overflow pages (squashed). 2018-09-23 18:07:29 +03:00
Leonid Yuriev
d2854e0760 mdbx: backport - refine mdbx_chk (squashed).
- refine 'mismatch idl length' error message.
 - add/fix printf-format checking.
 - refine dbi-structure.
2018-09-23 18:07:29 +03:00
Leonid Yuriev
5a29214ad9 mdbx-test: backport - update 'gc.sh' script (squashed). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
b51d92d449 mdbx-test: backport - update test (squashed).
- add support for 'default' options values.
 - add min/max cases for option values.
 - add support for db-geometry params.
 - fix int-types for 32-bit builds (minor).
 - fix key/value generation for long-length cases.
 - fix update_flags for non-MDBX_DUPSORT.
 - 'none' for config-verbs.
 - check commandline length under Windows.
 - workaround for QueryFullProcessImageNameA() bug.
 - add setloglevel().
 - workaroung for MSVC bug.
 - avoid extra 'jitter' testcase loops.
 - cleanup DUPSORT flags.
 - refine key/value min/max handling.
 - dump keygen params.
 - fix/refine keygen.
 - alter keygen defaults (rotate 3, offset 41).
 - default test-db size 4mb or 256mb.
 - fix/refine keygen for non-MDBX_DUPSORT.
 - seeding keygen with actor_id for better spreading.
2018-09-23 15:39:56 +03:00
Leonid Yuriev
6da477d37f mdbx-ci: backport - refines for Windows (squashed).
- push logs to appveyor separately.
 - rename 'test.exe' to 'mdbx_test.exe'.
 - add test.db to appveyor artefacts (windows).
2018-09-23 15:39:56 +03:00
Leonid Yuriev
6150a8c903 mdbx: backport - fix/refine mdbx_update_gc() (squashed). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
f3e9731da4 mdbx: backport - move macros/inlines to fix Windows builds.
Change-Id: I48aaf6b77466bb8b13294b84de73fb6063c88190
2018-09-23 15:39:56 +03:00
Leonid Yuriev
353b6b8af0 mdbx: backport - refine assections (minor).
Change-Id: Ic924988b8ce043d6106df381c996dd2c8ff9ca1f
2018-09-23 15:39:56 +03:00
Leonid Yuriev
3f10e58df2 mdbx: backport - re-define assert macro via mdbx_assert.
Change-Id: I317801ba4200bdf1aa5cacf75d21a8e633fbc48a
2018-09-23 15:39:56 +03:00
Leonid Yuriev
d232737087 mdbx: backport - add MDBX_FORCE_ASSERT.
Change-Id: I68a9f7b42663ea157c7c0a5a58797c94127b45ed
2018-09-23 15:39:56 +03:00
Leo Yuriev
e32ca55258 mdbx: backport - fix tracking around mdbx_cursor_del(). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
f57ffc987c mdbx: backport - drop inherited broken audit (will be fixed in the master branch).
Internal self-audit (inherited from LMDB) is invalid and useless
for sub-db and dupsort cases.
2018-09-23 15:39:56 +03:00
Leonid Yuriev
cdd510d20e mdbx: backport - prevent DB corruption due rebalance bugs.
Won't fix https://github.com/leo-yuriev/libmdbx/issues/38 in the 'stable/0.1' branch,
but add checks to prevent DB corruption.
2018-09-23 15:39:56 +03:00
Leo Yuriev
d757ba1266 mdbx: backport - fix MDBX_CORRUPTED due open/shrink collision. 2018-09-23 15:39:56 +03:00
Leonid Yuriev
337f7589f8 mdbx: backport - fix mdbx_pnl_search(). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
912728a322 mdbx: backport - fix mdbx_replace().
Change-Id: I2af00f101017795ca2b967479f86e5ea7e8ad37b
2018-09-23 15:39:56 +03:00
Leonid Yuriev
204b5a532d mdbx: backport - shorten maxkeysize (will be fixed in the master branch).
Change-Id: I660b1b3e454d9b51a24d3b4cc987c8e2980bd435
2018-09-23 15:39:56 +03:00
Leonid Yuriev
014be165c3 mdbx: backport - allow GC's PNL be partially unused. 2018-09-23 15:39:56 +03:00
Leonid Yuriev
a9244f807b mdbx: backport - setup mdbx_cmp_memn() as data-comparator for safety. 2018-09-23 15:39:56 +03:00
Leonid Yuriev
6d438605dd mdbx: backport - check comparator for MDBX_GET_BOTH and MDBX_GET_BOTH_RANGE. 2018-09-23 15:39:56 +03:00
Leo Yuriev
34300150a1 mdbx: backport - don't touch mm_psize and mm_flags while provoking bad readers (debug-only). 2018-09-23 15:39:56 +03:00
Leo Yuriev
1b2b98234f mdbx: backport - fix concurrent opening with custom pagesize (get pagesize from meta-page early). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
bd672a5583 mdbx: backport - add mdbx_limits_xyz() (squashed).
Change-Id: I56c79704c59386a0c4d84b001020484c23925e6c
2018-09-23 15:39:56 +03:00
Leonid Yuriev
888003c072 mdbx: backport - fix comments typos (squashed). 2018-09-23 14:40:31 +03:00
Leo Yuriev
bc6db4e4d7 mdbx: backport - allow mdbx_env_compact() to fix page leaks.
Don't treat fixing page leaks as an error while copy DB with compactification.

Change-Id: I2a575ff9e2b24610172aaca939b5f6957c26ec77
2018-09-23 14:40:31 +03:00
Leonid Yuriev
57655583e5 mdbx: backport - fix/rewrite mdbx_update_gc().
Change-Id: I580a1ff0cbeeb529e2bcbd50d97bfba7bcf5a546
2018-08-13 23:30:39 +03:00
Leonid Yuriev
b91e645919 mdbx-test: add 'gc.sh' script.
Change-Id: I633c93c0865b0d2609688713e986edf51ce6547d
2018-08-13 21:44:43 +03:00
Leonid Yuriev
652bb08f8c mdbx-test: backport - use strtoull() and retry with base=10.
Change-Id: Ica846ed0a13eb4468a45620518b9ccf85e77a764
2018-08-13 21:44:10 +03:00
Leonid Yuriev
59026d5f84 mdbx-test: backport - fix minor typos.
Change-Id: I4889a0e698bdfdda7eed257a5cd29e8b8089d102
2018-08-13 21:43:07 +03:00
Leonid Yuriev
e18551061e mdbx-test: backport - fix keylen/datalen min/max ranges checking.
Change-Id: Iee5d2f71ad22ec6e86167f5181deff54f0b5b518
2018-08-13 21:42:44 +03:00
Leonid Yuriev
e054ad2ebb mdbx-test: backport - add 'strikethrough' for bitmask-options.
Change-Id: I86dd2f8cdbd5a32a0471a5eee1e2b3a5857541ac
2018-08-13 21:42:22 +03:00
Leonid Yuriev
bff6aa460a mdbx: backport - fix MDBX_EKEYMISMATCH while update multi-value with MDBX_CURRENT.
Change-Id: I3095620a94f694fb2c29b9c4faab9ea02b9bd7b7
2018-08-13 21:41:17 +03:00
Leonid Yuriev
3979ba4784 mdbx: backport - fix assertions.
Change-Id: I95c43ef1ea2da55a124dc43f03890cf1d96f2e61
2018-08-13 21:40:48 +03:00
Leonid Yuriev
d2fcbf5f82 mdbx: backport - fix assert-condition inside mdbx_pnl_xappend().
Change-Id: Id5ac89c85b7e673c44d60a626c805fe666d221bc
2018-08-10 09:36:44 +03:00
Leo Yuriev
38067c4566 mdbx: backport - fix 'db_dummy' inside mdbx_dbi_open_ex().
Change-Id: I70a21c9b77a43c5af749da5723fa965487a056b0
2018-08-09 13:07:02 +03:00
Leo Yuriev
d4bfc17818 mdbx: backport - add fallback2shared for mdbx_lck_exclusive(). 2018-08-09 13:07:02 +03:00
Leo Yuriev
eb99480253 mdbx: backport - drop unused mdbx_lck_upgrade(). 2018-08-09 13:07:02 +03:00
Leo Yuriev
c9790b28d0 mdbx-cmake: fix so-version.
Change-Id: I427d2f27f9092d65a0ffd11353ca466070e98618
2018-08-09 13:07:02 +03:00
Leo Yuriev
9a1ef8acfb mdbx-cmake: remove warning-message.
Change-Id: Icf9e4f7a96916cf9ab04613344867217be04827a
2018-08-09 13:07:02 +03:00
Leo Yuriev
e442395cbd mdbx: bump version to v0.1.6
Change-Id: I95d45a815008e2cc9a8785a8c762310a1e907e21
2018-07-31 11:43:41 +03:00
Leo Yuriev
e57e521609 mdbx: backport - fix nasty suspend_and_append() bug.
Change-Id: I043adcff2e6c040426a51b5d4b15bac849e6dd9f
2018-07-31 11:43:35 +03:00
Leo Yuriev
d1809e6e2d mdbx: backport - minor fix to avoid Valgrind false-positive issue.
Change-Id: Ifa4dc51b500ff42a88182d750e22572aa5b2155b
2018-07-01 17:43:04 +03:00
Leo Yuriev
c579b974a2 mdbx: backport - avoid weak meta inside mdbx_init_metas().
Change-Id: Ib9c5ab04ad8cff3ad43d94a288cecec45d7ef37d
2018-06-30 02:43:10 +03:00
Leo Yuriev
de43ab0d21 mdbx-ci: migrate to Circle-CI 2.0
Change-Id: Id86af9e033d64a4dc2043db33cd8e7ae173feb22
2018-06-21 19:45:42 +03:00
Leo Yuriev
db50fb8726 mdbx: backport - fix Coverity warning (minor, paranoia).
Change-Id: I232377a03244dc33beb4f332c0024b454027f659
2018-06-21 18:26:31 +03:00
Leo Yuriev
408848a425 mdbx: Merge branch 'master' into stable/0.1
Change-Id: Ide2fbcd1b0b6bacbc4f07049633df81fede397eb
2018-06-21 17:57:06 +03:00
Leo Yuriev
685abc7bcb mdbx: more/again reformat by clang-format 6.0 (cosmetic).
Change-Id: I769ee5a80d75a49fc5bb041fefda6f22adc05424
2018-06-20 13:45:11 +03:00
Leo Yuriev
e880e734ce mdbx: Merge branch 'master' into stable/0.1 2018-06-19 14:46:39 +03:00
Leo Yuriev
e6c333c751 mdbx: add link to JNI bindings.
Change-Id: I8d39b06d2d433997a76e7d36c7da867eb66c3cf0
2018-06-18 23:04:56 +03:00
Leo Yuriev
9db090087e mdbx: reformat by clang-format 6.0 (cosmetic). 2018-06-18 21:29:12 +03:00
Leo Yuriev
b0550b38b8 mdbx-test: fix args quoting for Windows. 2018-06-18 18:17:13 +03:00
Leo Yuriev
93e3b4129a mdbx-tools: fix wrong 'bad sequence' error from mdbx_stat. 2018-06-18 16:36:53 +03:00
Leo Yuriev
61b2a7fc54 mdbx: Merge branch 'master' into stable/0.1
Change-Id: I80b5afe1d227009a42096b5c6f2bfa9e5eb09036
2018-06-15 03:46:17 +03:00
Leo Yuriev
f314cd6b92 mdbx: windows - fix truncation race while unmap.
Change-Id: I93983d100c78aa3e57c5a7ebd9d5bf2a96081ed7
2018-06-15 02:54:41 +03:00
Leo Yuriev
e1e17fd6a4 mdbx: Merge branch 'master' into stable/0.1 2018-06-14 13:54:01 +03:00
Leo Yuriev
b6e605b8da mdbx: bump version to v0.1.5 2018-06-14 13:53:05 +03:00
Leo Yuriev
2f983b281d mdbx: drop MDBX_DEVEL.
Change-Id: I5fc1a240e7909b8ecaf2ef19a83a9e59ea5609ba
2018-06-13 11:37:56 +03:00
Leo Yuriev
efdcbd8c35 mdbx: disable non-blocking DB-close under Windows.
Change-Id: If6579467132439b6b627e756f67f6bd35fed8b4f
2018-06-13 02:46:18 +03:00
Leo Yuriev
634efbe34b mdbx: don't block close_env() if other write-txn running.
Change-Id: I1d19afcc245578681b8bc3ee3966ac38bc596ec4
2018-06-12 13:18:17 +03:00
Leo Yuriev
17d3e7190c mdbx: Merge branch 'master' into stable/0.1
Change-Id: I7460e6a8b42c9bbeadfdf20d326c31c1d4a98969
2018-06-01 16:42:48 +03:00
Leo Yuriev
fdc248384e mdbx: skip meta if usedbytes beyond oef.
Change-Id: I1e95136bce7169b7ed612f9746d0cdec43caca14
2018-06-01 13:27:20 +03:00
Leo Yuriev
38369bd24b mdbx: force steady-sync when shrinking DB (fix corruption bug).
Change-Id: I636ee2ad04f28d4038d74bdf2c7061ab885f4b82
2018-06-01 13:15:21 +03:00
Leo Yuriev
22d8b0b2e1 mdbx: Merge branch 'master' into stable/0.1
Change-Id: Idd1af2c07b48bfb1f335aa31a52bed4bd68514ac
2018-05-29 03:14:07 +03:00
dartraiden
515adb674b mdbx: fix typos and punctuation 2018-05-29 03:00:48 +03:00
Leo Yuriev
f99df1ca35 mdbx: update .gitignore
Change-Id: I880b5ccca285394845300a600e2722a915f37cfe
2018-05-28 21:34:45 +03:00
Leo Yuriev
ab4c9c9db0 mdbx: return MDBX_CORRUPTED instead of crash if MDBX_DUPSORT mismatch.
Change-Id: I439a3c859dbfcfbe33a3db077c3e8200a1417c31
2018-05-28 20:04:48 +03:00
Leo Yuriev
b68ed8600c mdbx: more refine/fix README.
Change-Id: Icd26630d9d907fc5cdfd08b39a5226e78e982ffb
2018-05-25 00:00:58 +03:00
Leo Yuriev
9384d0efa6 mdbx: Merge branch 'master' into stable/0.1
Change-Id: I3318b330e4aa5fd0db4642af6ef81e69814f0cc5
2018-05-22 12:03:57 +03:00
Leo Yuriev
4214436d4b mdbx: fix/refine README.
Change-Id: I33817021ef5c74f0fd074d5ca6b61fd993e3d9f4
2018-05-22 12:01:35 +03:00
Leo Yuriev
ffafd5be10 mdbx: disable warning #5045 for MSVC (minor). 2018-05-21 16:36:41 +03:00
Leo Yuriev
e86bd88751 mdbx: disable warning #5045 for MSVC (minor). 2018-05-21 16:31:36 +03:00
Leo Yuriev
eb4090c534 mdbx: update Project Status. 2018-05-10 14:05:48 +03:00
Leo Yuriev
bea1349dd4 mdbx: bump internal version info to v0.1.4 2018-05-04 13:21:18 +03:00
Leo Yuriev
d0ddc8569b mdbx-windows: fix lck_reader_alive_check().
Change-Id: Icbc54d9b3003f70bba573a3807c5161380339460
2018-05-04 02:17:21 +03:00
Leo Yuriev
f1edd1579a mdbx: fix wrong freeDB search.
Avoid search freeDB while tree is updating.

This resolves https://github.com/leo-yuriev/libmdbx/issues/31
Bug was inherited from LMDB.

Change-Id: I0b12f8d9b88bca5aa26ca27db81df9b72c6a19b7
2018-04-25 03:01:12 +03:00
Leo Yuriev
47cc2b1c10 mdbx-cmake: fix version inside CMakeLists.txt
This resolves https://github.com/leo-yuriev/libmdbx/issues/32
2018-04-18 12:58:56 +03:00
Leo Yuriev
d127c28e83 mdbx-windows: fix srw-release if rdt-lock failed (minor/paranoia). 2018-04-12 00:56:58 +03:00
Leo Yuriev
159d676429 mdbx: fix typo in mdbx_chk output. 2018-04-10 21:12:41 +03:00
Leo Yuriev
f4a9018690 mdbx: bump internal version info to v0.1.3
Change-Id: I7db311c812d531f6628101715f6658005db3ea24
2018-04-03 20:37:07 +03:00
Leo Yuriev
f8eb423d36 mdbx-make: remote '-O3' for Elbrus (done by __hot attributes).
Change-Id: I6a3afeaf1ddf6d231941aae023dff89046bc4349
2018-04-03 20:06:45 +03:00
Leo Yuriev
797ae9d1db mdbx: minor fix debug-output for mdbx_thread_key_create().
Change-Id: I718130cf2385b6221187cbcdeedd9d48d05289bb
2018-04-03 20:04:10 +03:00
Leo Yuriev
f08c7ccac0 mdbx-make: rename mdbx_test and link with dso-library.
Change-Id: I39a04f82bc31c0865a8d02379a596048518205cd
2018-04-03 19:28:12 +03:00
Leo Yuriev
864be54f01 mdbx: fix/rework rthc counting.
Change-Id: I97fd5bd6dc1a46658138d82db10e937c9595d81f
2018-04-03 19:28:12 +03:00
Leo Yuriev
571b50622e mdbx: restore workaround for glibc's bug #21031.
Workaround for https://sourceware.org/bugzilla/show_bug.cgi?id=21031

Change-Id: I6cf9e037cc2fc298096b78ec96773f19478ed5c0
2018-04-03 17:44:18 +03:00
Leo Yuriev
36a86bbc6e mdbx: fix cursor tracking inside mdbx_rebalance(). 2018-03-26 20:20:14 +03:00
Leo Yuriev
ae708aab13 mdbx: update README.md 2018-03-26 14:17:35 +03:00
Leo Yuriev
02fc5fe158 mdbx: update internal version info. 2018-03-23 17:23:28 +03:00
Leo Yuriev
45f159ddd3 mdbx: mdbx_cursor_put(MDBX_APPEND+MDBX_NOOVERWRITE) should return MDBX_KEYEXIST instead of MDBX_EKEYMISMATCH.
When MDBX_APPEND is used the ordering of keys is important.
Therefore MDBX_EKEYMISMATCH should me returned when given key <= last.

On the other hand, if MDBX_NOOVERWRITE is also used
then the MDBX_KEYEXIST should be returned, but not MDBX_EKEYMISMATCH.
2018-03-23 17:21:27 +03:00
36 changed files with 2199 additions and 1291 deletions

20
.circleci/config.yml Normal file
View File

@@ -0,0 +1,20 @@
version: 2
jobs:
build:
docker:
- image: circleci/buildpack-deps:artful
environment:
- TESTDB: /tmp/test.db
- TESTLOG: /tmp/test.log
steps:
- checkout
- run: make all
- run: ulimit -c unlimited && make check
- run:
command: |
mkdir -p /tmp/artifacts
mv -t /tmp/artifacts $TESTLOG $TESTDB core.*
when: on_fail
- store_artifacts:
path: /tmp/artifacts
destination: test-artifacts

3
.gitignore vendored
View File

@@ -16,6 +16,7 @@ Win32/
build-*
cmake-build-*
core
example
libmdbx.creator.user
mdbx-dll.VC.VC.opendb
mdbx-dll.VC.db
@@ -25,7 +26,7 @@ mdbx_copy
mdbx_dump
mdbx_load
mdbx_stat
ootest/test
mdbx_test
test.log
test/test.vcxproj.user
test/tmp.db

View File

@@ -1,17 +1,13 @@
cmake_minimum_required(VERSION 2.8.7)
set(TARGET mdbx)
project(${TARGET})
# FIXME/TODO: Same as https://github.com/leo-yuriev/libfpta
set(MDBX_VERSION_MAJOR 0)
set(MDBX_VERSION_MINOR 0)
set(MDBX_VERSION_PATCH 0)
set(MDBX_VERSION_STRING ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}.${MDBX_VERSION_PATCH})
set(MDBX_VERSION_MINOR 1)
set(MDBX_VERSION_RELEASE 6)
set(MDBX_VERSION_REVISION 0)
add_definitions(-DMDBX_VERSION_MAJOR=${MDBX_VERSION_MAJOR})
add_definitions(-DMDBX_VERSION_MINOR=${MDBX_VERSION_MINOR})
add_definitions(-DMDBX_VERSION_PATCH=${MDBX_VERSION_PATCH})
set(MDBX_VERSION_STRING ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}.${MDBX_VERSION_RELEASE})
enable_language(C)
enable_language(CXX)
@@ -34,7 +30,6 @@ else()
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O2")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g3")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wall")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-constant-logical-operand")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Werror")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wextra")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -ffunction-sections")
@@ -107,14 +102,14 @@ add_library(${TARGET}_SHARED SHARED
set_target_properties(${TARGET}_SHARED PROPERTIES
VERSION ${MDBX_VERSION_STRING}
SOVERSION ${MDBX_VERSION_MAJOR}
SOVERSION ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}
OUTPUT_NAME ${TARGET}
CLEAN_DIRECT_OUTPUT 1
)
set_target_properties(${TARGET}_STATIC PROPERTIES
VERSION ${MDBX_VERSION_STRING}
SOVERSION ${MDBX_VERSION_MAJOR}
SOVERSION ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}
OUTPUT_NAME ${TARGET}
CLEAN_DIRECT_OUTPUT 1
)

View File

@@ -23,11 +23,7 @@ suffix ?=
CC ?= gcc
CXX ?= g++
ifeq ($(shell (export LC_ALL=C; ($(CC) --version 2>&1; $(CC) -v 2>&1) | grep -q -i 'e2k' && echo yes)),yes)
CFLAGS ?= -O3 -g3 -Wall -Werror -Wextra -ffunction-sections -fPIC -fvisibility=hidden
else
CFLAGS ?= -O2 -g3 -Wall -Werror -Wextra -ffunction-sections -fPIC -fvisibility=hidden
endif
XCFLAGS ?= -DNDEBUG=1 -DMDBX_DEBUG=0 -DLIBMDBX_EXPORTS=1
CFLAGS += -D_GNU_SOURCE=1 -std=gnu11 -pthread $(XCFLAGS)
@@ -37,7 +33,7 @@ TESTLOG ?= $(shell [ -d /dev/shm ] && echo /dev/shm || echo /tmp)/mdbx-test.log
# LY: '--no-as-needed,-lrt' for ability to built with modern glibc, but then run with the old
LDFLAGS ?= -Wl,--gc-sections,-z,relro,-O,--no-as-needed,-lrt
EXE_LDFLAGS ?= $(LDFLAGS) -static
EXE_LDFLAGS ?= -pthread -lrt
# LY: just for benchmarking
IOARENA ?= $(shell \
@@ -63,7 +59,7 @@ TEST_OBJ := $(patsubst %.cc,%.o,$(TEST_SRC))
.PHONY: mdbx all install clean check coverage
all: $(LIBRARIES) $(TOOLS) test/test example
all: $(LIBRARIES) $(TOOLS) mdbx_test example
mdbx: libmdbx.a libmdbx.so
@@ -83,16 +79,16 @@ install: $(LIBRARIES) $(TOOLS) $(HEADERS)
&& cp -t $(SANDBOX)$(mandir)/man1 $(MANPAGES)
clean:
rm -rf $(TOOLS) test/test @* *.[ao] *.[ls]o *~ tmp.db/* *.gcov *.log *.err src/*.o test/*.o
rm -rf $(TOOLS) mdbx_test @* *.[ao] *.[ls]o *~ tmp.db/* *.gcov *.log *.err src/*.o test/*.o
check: all
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; test/test --pathname=$(TESTDB) --dont-cleanup-after basic | tee -a $(TESTLOG) | tail -n 42) && ./mdbx_chk -vvn $(TESTDB)
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; ./mdbx_test --pathname=$(TESTDB) --dont-cleanup-after basic | tee -a $(TESTLOG) | tail -n 42) && ./mdbx_chk -vvn $(TESTDB)
check-singleprocess: all
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; test/test --pathname=$(TESTDB) --dont-cleanup-after --hill | tee -a $(TESTLOG) | tail -n 42) && ./mdbx_chk -vvn $(TESTDB)
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; ./mdbx_test --pathname=$(TESTDB) --dont-cleanup-after --hill | tee -a $(TESTLOG) | tail -n 42) && ./mdbx_chk -vvn $(TESTDB)
check-fault: all
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; test/test --pathname=$(TESTDB) --inject-writefault=42 --dump-config --dont-cleanup-after basic | tee -a $(TESTLOG) | tail -n 42) && ./mdbx_chk -vvn $(TESTDB)
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; ./mdbx_test --pathname=$(TESTDB) --inject-writefault=42 --dump-config --dont-cleanup-after basic | tee -a $(TESTLOG) | tail -n 42) && ./mdbx_chk -vvn $(TESTDB)
define core-rule
$(patsubst %.c,%.o,$(1)): $(1) $(CORE_INC) mdbx.h Makefile
@@ -117,8 +113,8 @@ libmdbx.so: $(CORE_OBJ)
mdbx_%: src/tools/mdbx_%.c libmdbx.a
$(CC) $(CFLAGS) $^ $(EXE_LDFLAGS) -o $@
test/test: $(TEST_OBJ) libmdbx.a
$(CXX) $(CXXFLAGS) $^ $(EXE_LDFLAGS) -o $@
mdbx_test: $(TEST_OBJ) libmdbx.so
$(CXX) $(CXXFLAGS) $(TEST_OBJ) -Wl,-rpath . -L . -l mdbx $(EXE_LDFLAGS) -o $@
###############################################################################

View File

@@ -12,19 +12,30 @@ and [by Yandex](https://translate.yandex.ru/translate?url=https%3A%2F%2Fgithub.c
### Project Status
**Now MDBX is under _active development_** and until 2018Q2 is expected a big
change both of API and database format. Unfortunately those update will lead to
loss of compatibility with previous versions.
The aim of this revolution in providing a clearer robust API and adding new
features, including the database properties.
**Сейчас MDBX _активно перерабатывается_** и к середине 2018
ожидается большое изменение как API, так и формата базы данных.
К сожалению, обновление приведет к потере совместимости с
предыдущими версиями.
Цель этой революции - обеспечение более четкого надежного
API и добавление новых функции, а также наделение базы данных
новыми свойствами.
В настоящее время MDBX предназначена для Linux, а также
поддерживает Windows (начиная с Windows Server 2008) в качестве
дополнительной платформы. Поддержка других ОС может быть
обеспечена на коммерческой основе. Однако такие
усовершенствования (т. е. pull-requests) могут быть приняты в
мейнстрим только в том случае, если будет доступен
соответствующий публичный и бесплатный сервис непрерывной
интеграции (aka Continuous Integration).
## Содержание
- [Обзор](#Обзор)
- [Сравнение с другими СУБД](#Сравнение-с-другими-СУБД)
- [История & Acknowledgements](#История)
- [История & Acknowledgments](#История)
- [Основные свойства](#Основные-свойства)
- [Сравнение производительности](#Сравнение-производительности)
- [Интегральная производительность](#Интегральная-производительность)
@@ -43,7 +54,7 @@ features, including the database properties.
_libmdbx_ - это встраиваемый key-value движок хранения со специфическим
набором свойств и возможностей, ориентированный на создание уникальных
легковесных решений с предельной производительностью.
легковесных решений с предельной производительностью под Linux и Windows.
_libmdbx_ позволяет множеству процессов совместно читать и обновлять
несколько key-value таблиц с соблюдением [ACID](https://ru.wikipedia.org/wiki/ACID),
@@ -51,8 +62,8 @@ _libmdbx_ позволяет множеству процессов совмес
_libmdbx_ обеспечивает
[serializability](https://en.wikipedia.org/wiki/Serializability)
изменений и согласованность данных после аварий. При этом транзакции
изменяющие данные никак не мешают операциям чтения и выполняются строго
изменений и согласованность данных после аварий. При этом транзакции,
изменяющие данные, никак не мешают операциям чтения и выполняются строго
последовательно с использованием единственного
[мьютекса](https://en.wikipedia.org/wiki/Mutual_exclusion).
@@ -90,7 +101,7 @@ Tables](https://github.com/leo-yuriev/libfpta), aka ["Позитивные
Technologies](https://www.ptsecurity.ru).
#### Acknowledgements
#### Acknowledgments
Howard Chu (Symas Corporation) - the author of LMDB,
from which originated the MDBX in 2015.
@@ -102,7 +113,7 @@ which was used for begin development of LMDB.
Основные свойства
=================
_libmdbx_ наследует все ключевые возможности и особенности от
_libmdbx_ наследует все ключевые возможности и особенности
своего прародителя [LMDB](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database),
но с устранением ряда описываемых далее проблем и архитектурных недочетов.
@@ -118,7 +129,7 @@ _libmdbx_ наследует все ключевые возможности и
[MVCC](https://ru.wikipedia.org/wiki/MVCC) и
[COW](https://ru.wikipedia.org/wiki/%D0%9A%D0%BE%D0%BF%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5_%D0%BF%D1%80%D0%B8_%D0%B7%D0%B0%D0%BF%D0%B8%D1%81%D0%B8).
Изменения строго последовательны и не блокируются чтением,
конфликты между транзакциями не возможны.
конфликты между транзакциями невозможны.
При этом гарантируется чтение только зафиксированных данных, см [relaxing serializability](https://en.wikipedia.org/wiki/Serializability).
4. Чтение и поиск [без блокировок](https://ru.wikipedia.org/wiki/%D0%9D%D0%B5%D0%B1%D0%BB%D0%BE%D0%BA%D0%B8%D1%80%D1%83%D1%8E%D1%89%D0%B0%D1%8F_%D1%81%D0%B8%D0%BD%D1%85%D1%80%D0%BE%D0%BD%D0%B8%D0%B7%D0%B0%D1%86%D0%B8%D1%8F),
@@ -182,7 +193,7 @@ github](https://github.com/pmwkaa/ioarena/tree/HL%2B%2B2015).
1. Такое сравнение не совсем правомочно, его следует делать с движками
ориентированными на хранение данных в памяти ([Tarantool](https://tarantool.io/), [Redis](https://redis.io/)).
2. Превосходство libmdbx становится еще более подавляющем, что мешает
2. Превосходство libmdbx становится еще более подавляющим, что мешает
восприятию информации.
![Comparison #1: Integral Performance](https://raw.githubusercontent.com/wiki/leo-yuriev/libmdbx/img/perf-slide-1.png)
@@ -206,7 +217,7 @@ github](https://github.com/pmwkaa/ioarena/tree/HL%2B%2B2015).
- Логарифмическая шкала справа и желтые интервальные отрезки
соответствуют времени выполнения транзакций. При этом каждый отрезок
показывает минимальное и максимальное время затраченное на выполнение
показывает минимальное и максимальное время, затраченное на выполнение
транзакций, а крестиком отмечено среднеквадратичное значение.
Выполняется **10.000 транзакций в режиме синхронной фиксации данных** на
@@ -232,7 +243,7 @@ github](https://github.com/pmwkaa/ioarena/tree/HL%2B%2B2015).
- Логарифмическая шкала справа и желтые интервальные отрезки
соответствуют времени выполнения транзакций. При этом каждый отрезок
показывает минимальное и максимальное время затраченное на выполнение
показывает минимальное и максимальное время, затраченное на выполнение
транзакций, а крестиком отмечено среднеквадратичное значение.
Выполняется **100.000 транзакций в режиме отложенной фиксации данных**
@@ -263,7 +274,7 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
- Логарифмическая шкала справа и желтые интервальные отрезки
соответствуют времени выполнения транзакций. При этом каждый отрезок
показывает минимальное и максимальное время затраченное на выполнение
показывает минимальное и максимальное время, затраченное на выполнение
транзакций, а крестиком отмечено среднеквадратичное значение.
Выполняется **1.000.000 транзакций в режиме асинхронной фиксации
@@ -272,7 +283,7 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
консистентны на момент завершения одной из транзакций, но допускается
потеря изменений из значительного количества последних транзакций. Во
всех движках при этом включался режим предполагающий минимальную
нагрузку на диск по-записи, и соответственно минимальную гарантию
нагрузку на диск по записи, и соответственно минимальную гарантию
сохранности данных. В _libmdbx_ при этом используется режим асинхронной
записи измененных страниц на диск посредством ядра ОС и системного
вызова [msync(MS_ASYNC)](https://linux.die.net/man/2/msync).
@@ -337,7 +348,7 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
> _libmdbx_ фиксация транзакции также требует от 2 IOPS. Однако, в БД с
> журналом кол-во IOPS будет меняться в зависимости от файловой системы,
> но не от кол-ва записей или их объема. Тогда как в _libmdbx_ кол-во
> будет расти логарифмически от кол-во записей/строк в БД (по высоте
> будет расти логарифмически от кол-ва записей/строк в БД (по высоте
> b+tree).
3. [COW](https://ru.wikipedia.org/wiki/%D0%9A%D0%BE%D0%BF%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5_%D0%BF%D1%80%D0%B8_%D0%B7%D0%B0%D0%BF%D0%B8%D1%81%D0%B8)
@@ -353,7 +364,7 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
> значительного копирования данных в памяти и массы других затратных операций.
> Поэтому обусловленное этим недостатком падение производительности становится
> заметным только при отказе от фиксации изменений на диске.
> Соответственно, корректнее сказать что _libmdbx_ позволяет
> Соответственно, корректнее сказать, что _libmdbx_ позволяет
> получить персистентность ценой минимального падения производительности.
> Если же нет необходимости оперативно сохранять данные, то логичнее
> использовать `std::map`.
@@ -446,7 +457,7 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
Однако, при аварийном отключении питания или сбое в ядре ОС, на диске
может быть сохранена только часть измененных страниц БД. При этом с большой
вероятностью может оказаться так, что будут сохранены мета-страницы со
вероятностью может оказаться, что будут сохранены мета-страницы со
ссылками на страницы с новыми версиями данных, но не сами новые данные.
В этом случае БД будет безвозвратна разрушена, даже если до аварии
производилась полная синхронизация данных (посредством
@@ -460,11 +471,11 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
с переносом изменений после фиксации данных.
* При завершении транзакций, в зависимости от состояния
синхронности данных между диском и оперативной память,
синхронности данных между диском и оперативной памятью,
_libmdbx_ помечает точки фиксации либо как сильные (strong),
либо как слабые (weak). Так например, в режиме
`WRITEMAP+MAPSYNC` завершаемые транзакции помечаются как
слабые, а при явной синхронизации данных как сильные.
слабые, а при явной синхронизации данных - как сильные.
* В _libmdbx_ поддерживается не две, а три отдельные мета-страницы.
Это позволяет выполнять фиксацию транзакций с формированием как
@@ -478,20 +489,20 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
сильной фиксации. Этим обеспечивается гарантия сохранности БД.
Такая гарантия надежности не дается бесплатно. Для
сохранности данных, страницы формирующие крайний снимок с
сохранности данных, страницы, формирующие крайний снимок с
сильной фиксацией, не должны повторно использоваться
(перезаписываться) до формирования следующей сильной точки
фиксации. Таким образом, крайняя точка фиксации создает
описанный выше эффект "долгого чтения". Разница же здесь в том,
что при исчерпании свободных страниц ситуация будет
автоматически исправлена, посредством записи изменений на диск
и формированием новой сильной точки фиксации.
и формирования новой сильной точки фиксации.
Таким образом, в режиме безопасной асинхронной фиксации _libmdbx_ будет
всегда использовать новые страницы до исчерпания места в БД или до явного
формирования сильной точки фиксации посредством `mdbx_env_sync()`.
При этом суммарный трафик записи на диск будет примерно такой-же,
как если бы отдельно фиксировалась каждая транзакций.
При этом суммарный трафик записи на диск будет примерно такой же,
как если бы отдельно фиксировалась каждая транзакция.
В текущей версии _libmdbx_ вам предоставляется выбор между безопасным
режимом (по умолчанию) асинхронной фиксации, и режимом `UTTERLY_NOSYNC` когда
@@ -524,9 +535,9 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
Посредством `mdbx_env_set_oomfunc()` может быть установлен
внешний обработчик (callback), который будет вызван при
исчерпания свободных страниц из-за долгой операцией чтения.
исчерпании свободных страниц из-за долгой операцией чтения.
Обработчику будет передан PID и pthread_id виновника.
В свою очередь обработчик может предпринять одно из действий:
В свою очередь обработчик может предпринять одно из действий:
* нейтрализовать виновника (отправить сигнал kill #9), если
долгое чтение выполняется сторонним процессом;
@@ -534,7 +545,7 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
* отменить или перезапустить проблемную операцию чтения, если
операция выполняется одним из потоков текущего процесса;
* подождать некоторое время, в расчете что проблемная операция
* подождать некоторое время, в расчете на то, что проблемная операция
чтения будет штатно завершена;
* прервать текущую операцию изменения данных с возвратом кода
@@ -601,7 +612,7 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
19. Возможность посредством `mdbx_is_dirty()` определить находятся ли
некоторый ключ или данные в "грязной" странице БД. Таким образом,
избегая лишнего копирования данных перед выполнением модифицирующих
операций (значения в размещенные "грязных" страницах могут быть
операций (значения, размещенные в "грязных" страницах, могут быть
перезаписаны при изменениях, иначе они будут неизменны).
20. Корректное обновление текущей записи, в том числе сортированного
@@ -621,7 +632,7 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
>
> - обращение к уже освобожденной памяти;
> - попытки повторного освобождения памяти;
> - memory corruption and segfaults.
> - повреждение памяти и ошибки сегментации.
22. Дополнительный код ошибки `MDBX_EMULTIVAL`, который возвращается из
`mdbx_put()` и `mdbx_replace()` при попытке выполнить неоднозначное
@@ -655,7 +666,7 @@ _libmdbx_ при этом не ведет WAL, а передает весь ко
цикле обновления данных и записи на диск. Фактически _libmdbx_ выполняет
постоянную компактификацию данных, но не затрачивая на это
дополнительных ресурсов, а только освобождая их. При освобождении места
в БД и установки соответствующих параметров геометрии базы данных, также будет
в БД и установке соответствующих параметров геометрии базы данных, также будет
уменьшаться размер файла на диске, в том числе в **Windows**.
--------------------------------------------------------------------------------

View File

@@ -7,17 +7,35 @@ libmdbx
[![Build status](https://ci.appveyor.com/api/projects/status/ue94mlopn50dqiqg/branch/master?svg=true)](https://ci.appveyor.com/project/leo-yuriev/libmdbx/branch/master)
[![Coverity Scan Status](https://scan.coverity.com/projects/12915/badge.svg)](https://scan.coverity.com/projects/reopen-libmdbx)
### Project Status
## Project Status for now
**MDBX is under _active development_**, database format and API aren't stable
at least until 2018Q2. New version won't be backwards compatible.
Main focus of the rework is to provide clear and robust API and new features.
- The stable versions ([_stable/0.0_](https://github.com/leo-yuriev/libmdbx/tree/stable/0.0) and [_stable/0.1_](https://github.com/leo-yuriev/libmdbx/tree/stable/0.1) branches) of _MDBX_ are frozen, i.e. no new features or API changes, but only bug fixes.
- The next version ([_devel_](https://github.com/leo-yuriev/libmdbx/tree/devel) branch) **is under active non-public development**, i.e. current API and set of features are extreme volatile.
- The immediate goal of development is formation of the stable API and the stable internal database format, which allows realise all PLANNED FEATURES:
1. Integrity check by [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree);
2. Support for [raw block devices](https://en.wikipedia.org/wiki/Raw_device);
3. Separate place (HDD) for large data items;
4. Using "[Roaring bitmaps](http://roaringbitmap.org/about/)" inside garbage collector;
5. Non-sequential reclaiming, like PostgreSQL's [Vacuum](https://www.postgresql.org/docs/9.1/static/sql-vacuum.html);
6. [Asynchronous lazy data flushing](https://sites.fas.harvard.edu/~cs265/papers/kathuria-2008.pdf) to disk(s);
7. etc...
Don't miss [Java Native Interface](https://github.com/castortech/mdbxjni) by [Castor Technologies](https://castortech.com/).
-----
Nowadays MDBX intended for Linux, and support Windows (since
Windows Server 2008) as a complementary platform. Support for
other OS could be implemented on commercial basis. However such
enhancements (i.e. pull requests) could be accepted in
mainstream only when corresponding public and free Continuous
Integration service will be available.
## Contents
- [Overview](#overview)
- [Comparison with other DBs](#comparison-with-other-dbs)
- [History & Acknowledgements](#history)
- [History & Acknowledgments](#history)
- [Main features](#main-features)
- [Performance comparison](#performance-comparison)
- [Integral performance](#integral-performance)
@@ -34,7 +52,7 @@ Main focus of the rework is to provide clear and robust API and new features.
## Overview
_libmdbx_ is an embedded lightweight key-value database engine oriented for performance.
_libmdbx_ is an embedded lightweight key-value database engine oriented for performance under Linux and Windows.
_libmdbx_ allows multiple processes to read and update several key-value tables concurrently,
while being [ACID](https://en.wikipedia.org/wiki/ACID)-compliant, with minimal overhead and operation cost of Olog(N).
@@ -57,7 +75,7 @@ Because _libmdbx_ is currently overhauled, I think it's better to just link
### History
_libmdbx_ design is based on [Lightning Memory-Mapped Database](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database).
The _libmdbx_ design is based on [Lightning Memory-Mapped Database](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database).
Initial development was going in [ReOpenLDAP](https://github.com/leo-yuriev/ReOpenLDAP) project, about a year later it
received separate development effort and in autumn 2015 was isolated to separate project, which was
[presented at Highload++ 2015 conference](http://www.highload.ru/2015/abstracts/1831.html).
@@ -65,7 +83,7 @@ received separate development effort and in autumn 2015 was isolated to separate
Since early 2017 _libmdbx_ is used in [Fast Positive Tables](https://github.com/leo-yuriev/libfpta),
by [Positive Technologies](https://www.ptsecurity.com).
#### Acknowledgements
#### Acknowledgments
Howard Chu (Symas Corporation) - the author of LMDB,
from which originated the MDBX in 2015.
@@ -261,7 +279,7 @@ scanning data directory.
#### Long-time read transactions problem
Garbage collection problem exists in all databases one way or another (e.g. VACUUM in PostgreSQL).
But in _libmbdx_ and LMDB it's even more important because of high performance and deliberate
But in _libmdbx_ and LMDB it's even more important because of high performance and deliberate
simplification of internals with emphasis on performance.
* Altering data during long read operation may exhaust available space on persistent storage.
@@ -297,7 +315,7 @@ _libmdbx_ addresses the problem, details below. Illustrations to this problem ca
In `WRITEMAP+MAPSYNC` mode dirty pages are written to persistent storage by kernel. This means that in case of application
crash OS kernel will write all dirty data to disk and nothing will be lost. But in case of hardware malfunction or OS kernel
fatal error only some dirty data might be synced to disk, and there is high probability that pages with metadata saved,
will point to non-saved, hence non-existent, data pages. In such situation DB is completely corrupted and can't be
will point to non-saved, hence non-existent, data pages. In such situation, DB is completely corrupted and can't be
repaired even if there was full sync before the crash via `mdbx_env_sync().
_libmdbx_ addresses this by fully reimplementing write path of data:
@@ -320,7 +338,7 @@ synchronization point. So last steady synchronization point creates "long-time r
of memory exhaustion the problem will be immediately addressed by flushing changes to persistent storage and forming new steady
synchronization point.
So in async-write mode _libmdbx_ will always use new pages until memory is exhausted or `mdbx_env_sync()`is invoked. Total
So in async-write mode _libmdbx_ will always use new pages until memory is exhausted or `mdbx_env_sync()` is invoked. Total
disk usage will be almost the same as in sync-write mode.
Current _libmdbx_ gives a choice of safe async-write mode (default) and `UTTERLY_NOSYNC` mode which may result in full DB
@@ -336,7 +354,7 @@ Improvements over LMDB
1. `LIFO RECLAIM` mode:
The newest pages are picked for reuse instead of the oldest.
This allows to minimize reclaim loop and make it execution time independent from total page count.
This allows to minimize reclaim loop and make it execution time independent of total page count.
This results in OS kernel cache mechanisms working with maximum efficiency.
In case of using disk controllers or storages with
@@ -348,7 +366,7 @@ Improvements over LMDB
`mdbx_env_set_oomfunc()` allows to set a callback, which will be called
in the event of memory exhausting during long-time read transaction.
Callback will be invoked with PID and pthread_id of offending thread as parameters.
Callback can do any of this things to remedy the problem:
Callback can do any of these things to remedy the problem:
* wait for read transaction to finish normally;
@@ -392,7 +410,7 @@ Improvements over LMDB
15. Ability to close DB in "dirty" state (without data flush and creation of steady synchronization point)
via `mdbx_env_close_ex()`.
16. Ability to get addition info, including number of the oldest snapshot of DB, which is used by one of the readers.
16. Ability to get additional info, including number of the oldest snapshot of DB, which is used by one of the readers.
Implemented via `mdbx_env_info()`.
17. `mdbx_del()` doesn't ignore additional argument (specifier) `data`
@@ -401,7 +419,7 @@ Improvements over LMDB
18. Ability to open dbi-table with simultaneous setup of comparators for keys and values, via `mdbx_dbi_open_ex()`.
19. Ability to find out if key or value are in dirty page. This may be useful to make a decision to avoid
19. Ability to find out if key or value is in dirty page. This may be useful to make a decision to avoid
excessive CoW before updates. Implemented via `mdbx_is_dirty()`.
20. Correct update of current record in `MDBX_CURRENT` mode of `mdbx_cursor_put()`, including sorted duplicated.
@@ -432,12 +450,12 @@ Improvements over LMDB
27. Advanced dynamic control over DB size, including ability to choose page size via `mdbx_env_set_geometry()`,
including on Windows.
28. Three meta-pages instead two, this allows to guarantee consistently update weak sync-points without risking to
28. Three meta-pages instead of two, this allows to guarantee consistently update weak sync-points without risking to
corrupt last steady sync-point.
29. Automatic reclaim of freed pages to specific reserved space in the end of database file. This lowers amount of pages,
29. Automatic reclaim of freed pages to specific reserved space at the end of database file. This lowers amount of pages,
loaded to memory, used in update/flush loop. In fact _libmdbx_ constantly performs compactification of data,
but doesn't use addition resources for that. Space reclaim of DB and setup of database geometry parameters also decreases
but doesn't use additional resources for that. Space reclaim of DB and setup of database geometry parameters also decreases
size of the database on disk, including on Windows.
--------------------------------------------------------------------------------

View File

@@ -70,7 +70,7 @@
- [x] Привести в порядок volatile.
- [x] контроль meta.mapsize.
- [x] переработка формата: заголовки страниц, meta, clk...
- [x] зачистка Doxygen и бесполезных коментариев.
- [x] зачистка Doxygen и бесполезных комментариев.
- [x] Добавить поле типа контрольной суммы.
- [x] Добавить поле/флаг размера pgno_t.
- [x] Поменять сигнатуры.

View File

@@ -1,4 +1,4 @@
version: 0.1.2.{build}
version: 0.1.6.{build}
environment:
matrix:
@@ -32,21 +32,23 @@ build_script:
test_script:
- ps: |
if (($env:PLATFORM -eq "x86") -and (Test-Path "C:\projects\libmdbx\Win32\$env:CONFIGURATION\test.exe" -PathType Leaf)) {
$test = "C:\projects\libmdbx\Win32\$env:CONFIGURATION\test.exe"
if (($env:PLATFORM -eq "x86") -and (Test-Path "C:\projects\libmdbx\Win32\$env:CONFIGURATION\mdbx_test.exe" -PathType Leaf)) {
$mdbx_test = "C:\projects\libmdbx\Win32\$env:CONFIGURATION\mdbx_test.exe"
$mdbx_chk = "C:\projects\libmdbx\Win32\$env:CONFIGURATION\mdbx_chk.exe"
} elseif (($env:PLATFORM -ne "ARM") -and ($env:PLATFORM -ne "ARM64")) {
$test = "C:\projects\libmdbx\$env:PLATFORM\$env:CONFIGURATION\test.exe"
$mdbx_test = "C:\projects\libmdbx\$env:PLATFORM\$env:CONFIGURATION\mdbx_test.exe"
$mdbx_chk = "C:\projects\libmdbx\$env:PLATFORM\$env:CONFIGURATION\mdbx_chk.exe"
} else {
$test = ""
$mdbx_test = ""
$mdbx_chk = ""
}
if ($test -ne "") {
& "$test" --pathname=tmp.db --dont-cleanup-after basic | Tee-Object -file test.log | Select-Object -last 42
& "$mdbx_chk" -nvv tmp.db | Tee-Object -file chk.log | Select-Object -last 42
if ($mdbx_test -ne "") {
& "$mdbx_test" --pathname=test.db --dont-cleanup-after basic | Tee-Object -file test.log | Select-Object -last 42
& "$mdbx_chk" -nvv test.db | Tee-Object -file chk.log | Select-Object -last 42
}
on_failure:
- ps: Push-AppveyorArtifact test.log chk.log
- ps: Push-AppveyorArtifact test.log
- ps: Push-AppveyorArtifact test.db
- ps: Push-AppveyorArtifact chk.log

View File

@@ -1,14 +0,0 @@
machine:
timezone:
Europe/Moscow
database:
override:
compile:
override:
- make all
test:
override:
- make check || mv test.log ${CIRCLE_ARTIFACTS}/

42
mdbx.h
View File

@@ -340,33 +340,33 @@ typedef int(MDBX_cmp_func)(const MDBX_val *a, const MDBX_val *b);
typedef enum MDBX_cursor_op {
MDBX_FIRST, /* Position at first key/data item */
MDBX_FIRST_DUP, /* MDBX_DUPSORT-only: Position at first data item
* of current key. */
* of current key. */
MDBX_GET_BOTH, /* MDBX_DUPSORT-only: Position at key/data pair. */
MDBX_GET_BOTH_RANGE, /* MDBX_DUPSORT-only: position at key, nearest data. */
MDBX_GET_CURRENT, /* Return key/data at current cursor position */
MDBX_GET_MULTIPLE, /* MDBX_DUPFIXED-only: Return key and up to a page of
* duplicate data items from current cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE.*/
MDBX_GET_MULTIPLE, /* MDBX_DUPFIXED-only: Return up to a page of duplicate
* data items from current cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE. */
MDBX_LAST, /* Position at last key/data item */
MDBX_LAST_DUP, /* MDBX_DUPSORT-only: Position at last data item
* of current key. */
* of current key. */
MDBX_NEXT, /* Position at next data item */
MDBX_NEXT_DUP, /* MDBX_DUPSORT-only: Position at next data item
* of current key. */
MDBX_NEXT_MULTIPLE, /* MDBX_DUPFIXED-only: Return key and up to a page of
* duplicate data items from next cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE. */
* of current key. */
MDBX_NEXT_MULTIPLE, /* MDBX_DUPFIXED-only: Return up to a page of duplicate
* data items from next cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE. */
MDBX_NEXT_NODUP, /* Position at first data item of next key */
MDBX_PREV, /* Position at previous data item */
MDBX_PREV_DUP, /* MDBX_DUPSORT-only: Position at previous data item
* of current key. */
* of current key. */
MDBX_PREV_NODUP, /* Position at last data item of previous key */
MDBX_SET, /* Position at specified key */
MDBX_SET_KEY, /* Position at specified key, return both key and data */
MDBX_SET_RANGE, /* Position at first key greater than or equal to
* specified key. */
* specified key. */
MDBX_PREV_MULTIPLE /* MDBX_DUPFIXED-only: Position at previous page and
* return key and up to a page of duplicate data items. */
* return up to a page of duplicate data items. */
} MDBX_cursor_op;
/* Return Codes
@@ -468,7 +468,7 @@ typedef struct MDBX_envinfo {
uint64_t lower; /* lower limit for datafile size */
uint64_t upper; /* upper limit for datafile size */
uint64_t current; /* current datafile size */
uint64_t shrink; /* shrink theshold for datafile */
uint64_t shrink; /* shrink threshold for datafile */
uint64_t grow; /* growth step for datafile */
} mi_geo;
uint64_t mi_mapsize; /* Size of the data memory map */
@@ -900,7 +900,7 @@ LIBMDBX_API int mdbx_env_set_maxdbs(MDBX_env *env, MDBX_dbi dbs);
*
* Returns The maximum size of a key we can write. */
LIBMDBX_API int mdbx_env_get_maxkeysize(MDBX_env *env);
LIBMDBX_API int mdbx_get_maxkeysize(size_t pagesize);
LIBMDBX_API int mdbx_get_maxkeysize(intptr_t pagesize);
/* Set application information associated with the MDBX_env.
*
@@ -966,7 +966,7 @@ LIBMDBX_API int mdbx_env_set_assert(MDBX_env *env, MDBX_assert_func *func);
* Returns A non-zero error value on failure and 0 on success, some
* possible errors are:
* - MDBX_PANIC - a fatal error occurred earlier and the environment
* must be shut down.
* must be shut down.
* - MDBX_MAP_RESIZED - another process wrote data beyond this MDBX_env's
* mapsize and this environment's map must be resized
* as well. See mdbx_env_set_mapsize().
@@ -1461,6 +1461,9 @@ LIBMDBX_API int mdbx_cursor_put(MDBX_cursor *cursor, MDBX_val *key,
/* Delete current key/data pair
*
* This function deletes the key/data pair to which the cursor refers.
* This does not invalidate the cursor, so operations such as MDBX_NEXT
* can still be used on it. Both MDBX_NEXT and MDBX_GET_CURRENT will return
* the same record after this operation.
*
* [in] cursor A cursor handle returned by mdbx_cursor_open()
* [in] flags Options for this operation. This parameter must be set to 0
@@ -1637,7 +1640,9 @@ typedef int MDBX_pgvisitor_func(uint64_t pgno, unsigned pgnumber, void *ctx,
LIBMDBX_API int mdbx_env_pgwalk(MDBX_txn *txn, MDBX_pgvisitor_func *visitor,
void *ctx);
typedef struct mdbx_canary { uint64_t x, y, z, v; } mdbx_canary;
typedef struct mdbx_canary {
uint64_t x, y, z, v;
} mdbx_canary;
LIBMDBX_API int mdbx_canary_put(MDBX_txn *txn, const mdbx_canary *canary);
LIBMDBX_API int mdbx_canary_get(MDBX_txn *txn, mdbx_canary *canary);
@@ -1671,6 +1676,11 @@ LIBMDBX_API int mdbx_is_dirty(const MDBX_txn *txn, const void *ptr);
LIBMDBX_API int mdbx_dbi_sequence(MDBX_txn *txn, MDBX_dbi dbi, uint64_t *result,
uint64_t increment);
LIBMDBX_API int mdbx_limits_pgsize_min(void);
LIBMDBX_API int mdbx_limits_pgsize_max(void);
LIBMDBX_API intptr_t mdbx_limits_dbsize_min(intptr_t pagesize);
LIBMDBX_API intptr_t mdbx_limits_dbsize_max(intptr_t pagesize);
/*----------------------------------------------------------------------------*/
/* attribute support functions for Nexenta */
typedef uint_fast64_t mdbx_attr_t;

View File

@@ -23,11 +23,6 @@
# undef NDEBUG
#endif
/* Features under development */
#ifndef MDBX_DEVEL
# define MDBX_DEVEL 0
#endif
/*----------------------------------------------------------------------------*/
/* Should be defined before any includes */
@@ -45,6 +40,9 @@
#if _MSC_VER > 1800
# pragma warning(disable : 4464) /* relative include path contains '..' */
#endif
#if _MSC_VER > 1913
# pragma warning(disable : 5045) /* Compiler will insert Spectre mitigation... */
#endif
#pragma warning(disable : 4710) /* 'xyz': function not inlined */
#pragma warning(disable : 4711) /* function 'xyz' selected for automatic inline expansion */
#pragma warning(disable : 4201) /* nonstandard extension used : nameless struct / union */
@@ -141,9 +139,9 @@
#define MDBX_MAGIC UINT64_C(/* 56-bit prime */ 0x59659DBDEF4C11)
/* The version number for a database's datafile format. */
#define MDBX_DATA_VERSION ((MDBX_DEVEL) ? 255 : 2)
#define MDBX_DATA_VERSION 2
/* The version number for a database's lockfile format. */
#define MDBX_LOCK_VERSION ((MDBX_DEVEL) ? 255 : 2)
#define MDBX_LOCK_VERSION 2
/* handle for the DB used to track free pages. */
#define FREE_DBI 0
@@ -162,15 +160,13 @@
* size up to 2^44 bytes, in case of 4K pages. */
typedef uint32_t pgno_t;
#define PRIaPGNO PRIu32
#define MAX_PAGENO ((pgno_t)UINT64_C(0xffffFFFFffff))
#define MAX_PAGENO UINT32_C(0x7FFFffff)
#define MIN_PAGENO NUM_METAS
/* A transaction ID. */
typedef uint64_t txnid_t;
#define PRIaTXN PRIi64
#if MDBX_DEVEL
#define MIN_TXNID (UINT64_MAX - UINT32_MAX)
#elif MDBX_DEBUG
#if MDBX_DEBUG
#define MIN_TXNID UINT64_C(0x100000000)
#else
#define MIN_TXNID UINT64_C(1)
@@ -371,19 +367,19 @@ typedef struct MDBX_page {
#define PAGEHDRSZ ((unsigned)offsetof(MDBX_page, mp_data))
/* The maximum size of a database page.
*
* It is 64K, but value-PAGEHDRSZ must fit in MDBX_page.mp_upper.
*
* MDBX will use database pages < OS pages if needed.
* That causes more I/O in write transactions: The OS must
* know (read) the whole page before writing a partial page.
*
* Note that we don't currently support Huge pages. On Linux,
* regular data files cannot use Huge pages, and in general
* Huge pages aren't actually pageable. We rely on the OS
* demand-pager to read our data and page it out when memory
* pressure from other processes is high. So until OSs have
* actual paging support for Huge pages, they're not viable. */
*
* It is 64K, but value-PAGEHDRSZ must fit in MDBX_page.mp_upper.
*
* MDBX will use database pages < OS pages if needed.
* That causes more I/O in write transactions: The OS must
* know (read) the whole page before writing a partial page.
*
* Note that we don't currently support Huge pages. On Linux,
* regular data files cannot use Huge pages, and in general
* Huge pages aren't actually pageable. We rely on the OS
* demand-pager to read our data and page it out when memory
* pressure from other processes is high. So until OSs have
* actual paging support for Huge pages, they're not viable. */
#define MAX_PAGESIZE 0x10000u
#define MIN_PAGESIZE 512u
@@ -393,9 +389,7 @@ typedef struct MDBX_page {
#else
#define MAX_MAPSIZE32 UINT32_C(0x7ff80000)
#endif
#define MAX_MAPSIZE64 \
((sizeof(pgno_t) > 4) ? UINT64_C(0x7fffFFFFfff80000) \
: MAX_PAGENO * (uint64_t)MAX_PAGESIZE)
#define MAX_MAPSIZE64 (MAX_PAGENO * (uint64_t)MAX_PAGESIZE)
#define MAX_MAPSIZE ((sizeof(size_t) < 8) ? MAX_MAPSIZE32 : MAX_MAPSIZE64)
@@ -412,7 +406,7 @@ typedef struct MDBX_lockinfo {
volatile uint32_t mti_envmode;
#ifdef MDBX_OSAL_LOCK
/* Mutex protecting write access to this table. */
/* Mutex protecting write-txn. */
union {
MDBX_OSAL_LOCK mti_wmutex;
uint8_t pad_mti_wmutex[MDBX_OSAL_LOCK_SIZE % sizeof(size_t)];
@@ -755,8 +749,8 @@ struct MDBX_env {
MDBX_PNL me_free_pgs;
/* ID2L of pages written during a write txn. Length MDBX_PNL_UM_SIZE. */
MDBX_ID2L me_dirtylist;
/* Max number of freelist items that can fit in a single overflow page */
unsigned me_maxfree_1pg;
/* Number of freelist items that can fit in a single overflow page */
unsigned me_maxgc_ov1page;
/* Max size of a node on a page */
unsigned me_nodemax;
unsigned me_maxkey_limit; /* max size of a key */
@@ -821,15 +815,12 @@ void mdbx_panic(const char *fmt, ...)
#define mdbx_assert_enabled() unlikely(mdbx_runtime_flags &MDBX_DBG_ASSERT)
#define mdbx_audit_enabled() unlikely(mdbx_runtime_flags &MDBX_DBG_AUDIT)
#define mdbx_debug_enabled(type) \
unlikely(mdbx_runtime_flags &(type & (MDBX_DBG_TRACE | MDBX_DBG_EXTRA)))
#else
#define mdbx_debug_enabled(type) (0)
#define mdbx_audit_enabled() (0)
#ifndef NDEBUG
#if !defined(NDEBUG) || defined(MDBX_FORCE_ASSERT)
#define mdbx_assert_enabled() (1)
#else
#define mdbx_assert_enabled() (0)
@@ -935,44 +926,17 @@ void mdbx_panic(const char *fmt, ...)
/* assert(3) variant in transaction context */
#define mdbx_tassert(txn, expr) mdbx_assert((txn)->mt_env, expr)
static __inline void mdbx_jitter4testing(bool tiny) {
#ifndef NDEBUG
if (MDBX_DBG_JITTER & mdbx_runtime_flags)
mdbx_osal_jitter(tiny);
#else
(void)tiny;
#endif
}
/*----------------------------------------------------------------------------*/
/* Internal prototypes and inlines */
/* Internal prototypes */
int mdbx_reader_check0(MDBX_env *env, int rlocked, int *dead);
void mdbx_rthc_dtor(void *rthc);
void mdbx_rthc_lock(void);
void mdbx_rthc_unlock(void);
int mdbx_rthc_alloc(mdbx_thread_key_t *key, MDBX_reader *begin,
MDBX_reader *end);
void mdbx_rthc_remove(mdbx_thread_key_t key);
void mdbx_rthc_cleanup(void);
void mdbx_rthc_remove(const mdbx_thread_key_t key);
static __inline bool mdbx_is_power2(size_t x) { return (x & (x - 1)) == 0; }
static __inline size_t mdbx_roundup2(size_t value, size_t granularity) {
assert(mdbx_is_power2(granularity));
return (value + granularity - 1) & ~(granularity - 1);
}
static __inline unsigned mdbx_log2(size_t value) {
assert(mdbx_is_power2(value));
unsigned log = 0;
while (value > 1) {
log += 1;
value >>= 1;
}
return log;
}
void mdbx_rthc_global_init(void);
void mdbx_rthc_global_dtor(void);
void mdbx_rthc_thread_dtor(void *ptr);
#define MDBX_IS_ERROR(rc) \
((rc) != MDBX_RESULT_TRUE && (rc) != MDBX_RESULT_FALSE)
@@ -1047,6 +1011,8 @@ static __inline unsigned mdbx_log2(size_t value) {
/* Test if a page is a sub page */
#define IS_SUBP(p) F_ISSET((p)->mp_flags, P_SUBP)
#define PAGETYPE(p) ((p)->mp_flags & (P_BRANCH | P_LEAF | P_LEAF2 | P_OVERFLOW))
/* The number of overflow pages needed to store the given size. */
#define OVPAGES(env, size) (bytes2pgno(env, PAGEHDRSZ - 1 + (size)) + 1)
@@ -1117,71 +1083,12 @@ typedef struct MDBX_node {
* This is node header plus key plus data size. */
#define LEAFSIZE(k, d) (NODESIZE + (k)->iov_len + (d)->iov_len)
/* Address of node i in page p */
static __inline MDBX_node *NODEPTR(MDBX_page *p, unsigned i) {
assert(NUMKEYS(p) > (unsigned)(i));
return (MDBX_node *)((char *)(p) + (p)->mp_ptrs[i] + PAGEHDRSZ);
}
/* Address of the key for the node */
#define NODEKEY(node) (void *)((node)->mn_data)
/* Address of the data for a node */
#define NODEDATA(node) (void *)((char *)(node)->mn_data + (node)->mn_ksize)
/* Get the page number pointed to by a branch node */
static __inline pgno_t NODEPGNO(const MDBX_node *node) {
pgno_t pgno;
if (UNALIGNED_OK) {
pgno = node->mn_ksize_and_pgno;
if (sizeof(pgno_t) > 4)
pgno &= MAX_PAGENO;
} else {
pgno = node->mn_lo | ((pgno_t)node->mn_hi << 16);
if (sizeof(pgno_t) > 4)
pgno |= ((uint64_t)node->mn_flags) << 32;
}
return pgno;
}
/* Set the page number in a branch node */
static __inline void SETPGNO(MDBX_node *node, pgno_t pgno) {
assert(pgno <= MAX_PAGENO);
if (UNALIGNED_OK) {
if (sizeof(pgno_t) > 4)
pgno |= ((uint64_t)node->mn_ksize) << 48;
node->mn_ksize_and_pgno = pgno;
} else {
node->mn_lo = (uint16_t)pgno;
node->mn_hi = (uint16_t)(pgno >> 16);
if (sizeof(pgno_t) > 4)
node->mn_flags = (uint16_t)((uint64_t)pgno >> 32);
}
}
/* Get the size of the data in a leaf node */
static __inline size_t NODEDSZ(const MDBX_node *node) {
size_t size;
if (UNALIGNED_OK) {
size = node->mn_dsize;
} else {
size = node->mn_lo | ((size_t)node->mn_hi << 16);
}
return size;
}
/* Set the size of the data for a leaf node */
static __inline void SETDSZ(MDBX_node *node, size_t size) {
assert(size < INT_MAX);
if (UNALIGNED_OK) {
node->mn_dsize = (uint32_t)size;
} else {
node->mn_lo = (uint16_t)size;
node->mn_hi = (uint16_t)(size >> 16);
}
}
/* The size of a key in a node */
#define NODEKSZ(node) ((node)->mn_ksize)
@@ -1234,19 +1141,8 @@ static __inline void SETDSZ(MDBX_node *node, size_t size) {
#define mdbx_cmp2int(a, b) (((a) > (b)) - ((b) > (a)))
#endif
static __inline size_t pgno2bytes(const MDBX_env *env, pgno_t pgno) {
mdbx_assert(env, (1u << env->me_psize2log) == env->me_psize);
return ((size_t)pgno) << env->me_psize2log;
}
static __inline MDBX_page *pgno2page(const MDBX_env *env, pgno_t pgno) {
return (MDBX_page *)(env->me_map + pgno2bytes(env, pgno));
}
static __inline pgno_t bytes2pgno(const MDBX_env *env, size_t bytes) {
mdbx_assert(env, (env->me_psize >> env->me_psize2log) == 1);
return (pgno_t)(bytes >> env->me_psize2log);
}
/* Do not spill pages to disk if txn is getting full, may fail instead */
#define MDBX_NOSPILL 0x8000
static __inline pgno_t pgno_add(pgno_t base, pgno_t augend) {
assert(base <= MAX_PAGENO);
@@ -1258,10 +1154,11 @@ static __inline pgno_t pgno_sub(pgno_t base, pgno_t subtrahend) {
return (subtrahend < base - MIN_PAGENO) ? base - subtrahend : MIN_PAGENO;
}
static __inline size_t pgno_align2os_bytes(const MDBX_env *env, pgno_t pgno) {
return mdbx_roundup2(pgno2bytes(env, pgno), env->me_os_psize);
}
static __inline pgno_t pgno_align2os_pgno(const MDBX_env *env, pgno_t pgno) {
return bytes2pgno(env, pgno_align2os_bytes(env, pgno));
static __inline void mdbx_jitter4testing(bool tiny) {
#ifndef NDEBUG
if (MDBX_DBG_JITTER & mdbx_runtime_flags)
mdbx_osal_jitter(tiny);
#else
(void)tiny;
#endif
}

View File

@@ -32,18 +32,12 @@
/*----------------------------------------------------------------------------*/
/* rthc */
static pthread_mutex_t mdbx_rthc_mutex = PTHREAD_MUTEX_INITIALIZER;
void mdbx_rthc_lock(void) {
mdbx_ensure(NULL, pthread_mutex_lock(&mdbx_rthc_mutex) == 0);
static __cold __attribute__((constructor)) void mdbx_global_constructor(void) {
mdbx_rthc_global_init();
}
void mdbx_rthc_unlock(void) {
mdbx_ensure(NULL, pthread_mutex_unlock(&mdbx_rthc_mutex) == 0);
}
void __attribute__((destructor)) mdbx_global_destructor(void) {
mdbx_rthc_cleanup();
static __cold __attribute__((destructor)) void mdbx_global_destructor(void) {
mdbx_rthc_global_dtor();
}
/*----------------------------------------------------------------------------*/
@@ -74,11 +68,19 @@ static int mdbx_lck_op(mdbx_filehandle_t fd, int op, int lck, off_t offset,
}
}
static __inline int mdbx_lck_exclusive(int lfd) {
static __inline int mdbx_lck_exclusive(int lfd, bool fallback2shared) {
assert(lfd != INVALID_HANDLE_VALUE);
if (flock(lfd, LOCK_EX | LOCK_NB))
return errno;
return mdbx_lck_op(lfd, F_SETLK, F_WRLCK, 0, 1);
int rc = mdbx_lck_op(lfd, F_SETLK, F_WRLCK, 0, 1);
if (rc != 0 && fallback2shared) {
while (flock(lfd, LOCK_SH)) {
int rc = errno;
if (rc != EINTR)
return rc;
}
}
return rc;
}
static __inline int mdbx_lck_shared(int lfd) {
@@ -95,8 +97,6 @@ int mdbx_lck_downgrade(MDBX_env *env, bool complete) {
return complete ? mdbx_lck_shared(env->me_lfd) : MDBX_SUCCESS;
}
int mdbx_lck_upgrade(MDBX_env *env) { return mdbx_lck_exclusive(env->me_lfd); }
int mdbx_rpid_set(MDBX_env *env) {
return mdbx_lck_op(env->me_lfd, F_SETLK, F_WRLCK, env->me_pid, 1);
}
@@ -165,7 +165,7 @@ bailout:
void mdbx_lck_destroy(MDBX_env *env) {
if (env->me_lfd != INVALID_HANDLE_VALUE) {
/* try get exclusive access */
if (env->me_lck && mdbx_lck_exclusive(env->me_lfd) == 0) {
if (env->me_lck && mdbx_lck_exclusive(env->me_lfd, false) == 0) {
mdbx_info("%s: got exclusive, drown mutexes", mdbx_func_);
int rc = pthread_mutex_destroy(&env->me_lck->mti_rmutex);
if (rc == 0)
@@ -233,7 +233,7 @@ static int internal_seize_lck(int lfd) {
assert(lfd != INVALID_HANDLE_VALUE);
/* try exclusive access */
int rc = mdbx_lck_exclusive(lfd);
int rc = mdbx_lck_exclusive(lfd, false);
if (rc == 0)
/* got exclusive */
return MDBX_RESULT_TRUE;
@@ -242,7 +242,7 @@ static int internal_seize_lck(int lfd) {
rc = mdbx_lck_shared(lfd);
if (rc == 0) {
/* got shared, try exclusive again */
rc = mdbx_lck_exclusive(lfd);
rc = mdbx_lck_exclusive(lfd, true);
if (rc == 0)
/* now got exclusive */
return MDBX_RESULT_TRUE;

View File

@@ -27,31 +27,24 @@
/*----------------------------------------------------------------------------*/
/* rthc */
static CRITICAL_SECTION rthc_critical_section;
static void NTAPI tls_callback(PVOID module, DWORD reason, PVOID reserved) {
(void)module;
(void)reserved;
switch (reason) {
case DLL_PROCESS_ATTACH:
InitializeCriticalSection(&rthc_critical_section);
mdbx_rthc_global_init();
break;
case DLL_PROCESS_DETACH:
DeleteCriticalSection(&rthc_critical_section);
mdbx_rthc_global_dtor();
break;
case DLL_THREAD_ATTACH:
break;
case DLL_THREAD_DETACH:
mdbx_rthc_cleanup();
mdbx_rthc_thread_dtor(module);
break;
}
}
void mdbx_rthc_lock(void) { EnterCriticalSection(&rthc_critical_section); }
void mdbx_rthc_unlock(void) { LeaveCriticalSection(&rthc_critical_section); }
/* *INDENT-OFF* */
/* clang-format off */
#if defined(_MSC_VER)
@@ -134,8 +127,9 @@ int mdbx_txn_lock(MDBX_env *env, bool dontwait) {
EnterCriticalSection(&env->me_windowsbug_lock);
}
if (flock(env->me_fd, dontwait ? (LCK_EXCLUSIVE | LCK_DONTWAIT)
: (LCK_EXCLUSIVE | LCK_WAITFOR),
if (flock(env->me_fd,
dontwait ? (LCK_EXCLUSIVE | LCK_DONTWAIT)
: (LCK_EXCLUSIVE | LCK_WAITFOR),
LCK_BODY))
return MDBX_SUCCESS;
int rc = GetLastError();
@@ -169,7 +163,10 @@ int mdbx_rdt_lock(MDBX_env *env) {
/* transite from S-? (used) to S-E (locked), e.g. exclusive lock upper-part */
if (flock(env->me_lfd, LCK_EXCLUSIVE | LCK_WAITFOR, LCK_UPPER))
return MDBX_SUCCESS;
return GetLastError();
int rc = GetLastError();
ReleaseSRWLockShared(&env->me_remap_guard);
return rc;
}
void mdbx_rdt_unlock(MDBX_env *env) {
@@ -193,7 +190,9 @@ static int suspend_and_append(mdbx_handle_array_t **array,
(limit * 2 - ARRAY_LENGTH((*array)->handles)));
if (!ptr)
return MDBX_ENOMEM;
(*array) = (mdbx_handle_array_t *)ptr;
if (limit == ARRAY_LENGTH((*array)->handles))
memcpy(ptr, *array, sizeof(mdbx_handle_array_t));
*array = (mdbx_handle_array_t *)ptr;
(*array)->limit = limit * 2;
}
@@ -357,7 +356,7 @@ static int internal_seize_lck(HANDLE lfd) {
"?-E(middle) >> S-E(locked)", rc);
/* 8) now on S-E (locked) or still on ?-E (middle),
* transite to S-? (used) or ?-? (free) */
* transite to S-? (used) or ?-? (free) */
if (!funlock(lfd, LCK_UPPER))
mdbx_panic("%s(%s) failed: errcode %u", mdbx_func_,
"X-E(locked/middle) >> X-?(used/free)", GetLastError());
@@ -442,47 +441,6 @@ int mdbx_lck_downgrade(MDBX_env *env, bool complete) {
return MDBX_SUCCESS /* 7) now at S-? (used), done */;
}
int mdbx_lck_upgrade(MDBX_env *env) {
/* Transite from locked state (S-E) to exclusive-write (E-E) */
assert(env->me_fd != INVALID_HANDLE_VALUE);
assert(env->me_lfd != INVALID_HANDLE_VALUE);
/* 1) must be at S-E (locked), transite to ?_E (middle) */
if (!funlock(env->me_lfd, LCK_LOWER))
mdbx_panic("%s(%s) failed: errcode %u", mdbx_func_,
"S-E(locked) >> ?-E(middle)", GetLastError());
/* 3) now on ?-E (middle), try E-E (exclusive-write) */
mdbx_jitter4testing(false);
if (flock(env->me_lfd, LCK_EXCLUSIVE | LCK_DONTWAIT, LCK_LOWER))
return MDBX_RESULT_TRUE; /* 4) got E-E (exclusive-write), done */
/* 5) still on ?-E (middle) */
int rc = GetLastError();
mdbx_jitter4testing(false);
if (rc != ERROR_SHARING_VIOLATION && rc != ERROR_LOCK_VIOLATION) {
/* 6) something went wrong, report but continue */
mdbx_error("%s(%s) failed: errcode %u", mdbx_func_,
"?-E(middle) >> E-E(exclusive-write)", rc);
}
/* 7) still on ?-E (middle), try restore S-E (locked) */
mdbx_jitter4testing(false);
rc = flock(env->me_lfd, LCK_SHARED | LCK_DONTWAIT, LCK_LOWER)
? MDBX_RESULT_FALSE
: GetLastError();
mdbx_jitter4testing(false);
if (rc != MDBX_RESULT_FALSE) {
mdbx_fatal("%s(%s) failed: errcode %u", mdbx_func_,
"?-E(middle) >> S-E(locked)", rc);
return rc;
}
/* 8) now on S-E (locked) */
return MDBX_RESULT_FALSE;
}
void mdbx_lck_destroy(MDBX_env *env) {
int rc;
@@ -550,10 +508,12 @@ int mdbx_rpid_clear(MDBX_env *env) {
* or otherwise the errcode. */
int mdbx_rpid_check(MDBX_env *env, mdbx_pid_t pid) {
(void)env;
HANDLE hProcess = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);
HANDLE hProcess = OpenProcess(SYNCHRONIZE, FALSE, pid);
int rc;
if (hProcess) {
if (likely(hProcess)) {
rc = WaitForSingleObject(hProcess, 0);
if (unlikely(rc == WAIT_FAILED))
rc = GetLastError();
CloseHandle(hProcess);
} else {
rc = GetLastError();

1962
src/mdbx.c

File diff suppressed because it is too large Load Diff

View File

@@ -700,39 +700,6 @@ int mdbx_ftruncate(mdbx_filehandle_t fd, uint64_t length) {
/*----------------------------------------------------------------------------*/
int mdbx_thread_key_create(mdbx_thread_key_t *key) {
#if defined(_WIN32) || defined(_WIN64)
*key = TlsAlloc();
return (*key != TLS_OUT_OF_INDEXES) ? MDBX_SUCCESS : GetLastError();
#else
return pthread_key_create(key, mdbx_rthc_dtor);
#endif
}
void mdbx_thread_key_delete(mdbx_thread_key_t key) {
#if defined(_WIN32) || defined(_WIN64)
mdbx_ensure(NULL, TlsFree(key));
#else
mdbx_ensure(NULL, pthread_key_delete(key) == 0);
#endif
}
void *mdbx_thread_rthc_get(mdbx_thread_key_t key) {
#if defined(_WIN32) || defined(_WIN64)
return TlsGetValue(key);
#else
return pthread_getspecific(key);
#endif
}
void mdbx_thread_rthc_set(mdbx_thread_key_t key, const void *value) {
#if defined(_WIN32) || defined(_WIN64)
mdbx_ensure(NULL, TlsSetValue(key, (void *)value));
#else
mdbx_ensure(NULL, pthread_setspecific(key, value) == 0);
#endif
}
int mdbx_thread_create(mdbx_thread_t *thread,
THREAD_RESULT(THREAD_CALL *start_routine)(void *),
void *arg) {
@@ -870,13 +837,14 @@ int mdbx_mmap(int flags, mdbx_mmap_t *map, size_t size, size_t limit) {
SectionSize.QuadPart = size;
rc = NtCreateSection(
&map->section,
/* DesiredAccess */ (flags & MDBX_WRITEMAP)
/* DesiredAccess */
(flags & MDBX_WRITEMAP)
? SECTION_QUERY | SECTION_MAP_READ | SECTION_EXTEND_SIZE |
SECTION_MAP_WRITE
: SECTION_QUERY | SECTION_MAP_READ | SECTION_EXTEND_SIZE,
/* ObjectAttributes */ NULL, /* MaximumSize (InitialSize) */ &SectionSize,
/* SectionPageProtection */ (flags & MDBX_RDONLY) ? PAGE_READONLY
: PAGE_READWRITE,
/* SectionPageProtection */
(flags & MDBX_RDONLY) ? PAGE_READONLY : PAGE_READWRITE,
/* AllocationAttributes */ SEC_RESERVE, map->fd);
if (!NT_SUCCESS(rc))
return ntstatus2errcode(rc);
@@ -889,8 +857,8 @@ int mdbx_mmap(int flags, mdbx_mmap_t *map, size_t size, size_t limit) {
/* SectionOffset */ NULL, &ViewSize,
/* InheritDisposition */ ViewUnmap,
/* AllocationType */ (flags & MDBX_RDONLY) ? 0 : MEM_RESERVE,
/* Win32Protect */ (flags & MDBX_WRITEMAP) ? PAGE_READWRITE
: PAGE_READONLY);
/* Win32Protect */
(flags & MDBX_WRITEMAP) ? PAGE_READWRITE : PAGE_READONLY);
if (!NT_SUCCESS(rc)) {
NtClose(map->section);
map->section = 0;
@@ -925,11 +893,6 @@ int mdbx_munmap(mdbx_mmap_t *map) {
if (!NT_SUCCESS(rc))
ntstatus2errcode(rc);
if (map->filesize != map->current &&
mdbx_filesize(map->fd, &map->filesize) == MDBX_SUCCESS &&
map->filesize != map->current)
(void)mdbx_ftruncate(map->fd, map->current);
map->length = 0;
map->current = 0;
map->address = nullptr;
@@ -955,8 +918,11 @@ int mdbx_mresize(int flags, mdbx_mmap_t *map, size_t size, size_t limit) {
/* growth rw-section */
SectionSize.QuadPart = size;
status = NtExtendSection(map->section, &SectionSize);
if (NT_SUCCESS(status))
map->filesize = map->current = size;
if (NT_SUCCESS(status)) {
map->current = size;
if (map->filesize < size)
map->filesize = size;
}
return ntstatus2errcode(status);
}
@@ -976,10 +942,10 @@ int mdbx_mresize(int flags, mdbx_mmap_t *map, size_t size, size_t limit) {
}
/* Windows unable:
* - shrink a mapped file;
* - change size of mapped view;
* - extend read-only mapping;
* Therefore we should unmap/map entire section. */
* - shrink a mapped file;
* - change size of mapped view;
* - extend read-only mapping;
* Therefore we should unmap/map entire section. */
status = NtUnmapViewOfSection(GetCurrentProcess(), map->address);
if (!NT_SUCCESS(status))
return ntstatus2errcode(status);
@@ -1031,14 +997,15 @@ retry_file_and_section:
SectionSize.QuadPart = size;
status = NtCreateSection(
&map->section,
/* DesiredAccess */ (flags & MDBX_WRITEMAP)
/* DesiredAccess */
(flags & MDBX_WRITEMAP)
? SECTION_QUERY | SECTION_MAP_READ | SECTION_EXTEND_SIZE |
SECTION_MAP_WRITE
: SECTION_QUERY | SECTION_MAP_READ | SECTION_EXTEND_SIZE,
/* ObjectAttributes */ NULL,
/* MaximumSize (InitialSize) */ &SectionSize,
/* SectionPageProtection */ (flags & MDBX_RDONLY) ? PAGE_READONLY
: PAGE_READWRITE,
/* SectionPageProtection */
(flags & MDBX_RDONLY) ? PAGE_READONLY : PAGE_READWRITE,
/* AllocationAttributes */ SEC_RESERVE, map->fd);
if (!NT_SUCCESS(status))
@@ -1062,8 +1029,8 @@ retry_mapview:;
/* SectionOffset */ NULL, &ViewSize,
/* InheritDisposition */ ViewUnmap,
/* AllocationType */ (flags & MDBX_RDONLY) ? 0 : MEM_RESERVE,
/* Win32Protect */ (flags & MDBX_WRITEMAP) ? PAGE_READWRITE
: PAGE_READONLY);
/* Win32Protect */
(flags & MDBX_WRITEMAP) ? PAGE_READWRITE : PAGE_READONLY);
if (!NT_SUCCESS(status)) {
if (status == /* STATUS_CONFLICTING_ADDRESSES */ 0xC0000018 &&

View File

@@ -469,10 +469,6 @@ int mdbx_thread_create(mdbx_thread_t *thread,
THREAD_RESULT(THREAD_CALL *start_routine)(void *),
void *arg);
int mdbx_thread_join(mdbx_thread_t thread);
int mdbx_thread_key_create(mdbx_thread_key_t *key);
void mdbx_thread_key_delete(mdbx_thread_key_t key);
void *mdbx_thread_rthc_get(mdbx_thread_key_t key);
void mdbx_thread_rthc_set(mdbx_thread_key_t key, const void *value);
int mdbx_filesync(mdbx_filehandle_t fd, bool fullsync);
int mdbx_filesize_sync(mdbx_filehandle_t fd);
@@ -552,7 +548,6 @@ int mdbx_lck_init(MDBX_env *env);
int mdbx_lck_seize(MDBX_env *env);
int mdbx_lck_downgrade(MDBX_env *env, bool complete);
int mdbx_lck_upgrade(MDBX_env *env);
void mdbx_lck_destroy(MDBX_env *env);
int mdbx_rdt_lock(MDBX_env *env);

View File

@@ -1,4 +1,4 @@
/* mdbx_chk.c - memory-mapped database check tool */
/* mdbx_chk.c - memory-mapped database check tool */
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
@@ -61,12 +61,18 @@ static void signal_handler(int sig) {
#define EXIT_FAILURE_CHECK_MAJOR (EXIT_FAILURE + 1)
#define EXIT_FAILURE_CHECK_MINOR EXIT_FAILURE
typedef struct {
const char *name;
struct {
uint64_t total;
uint64_t empty;
} pages;
uint64_t payload_bytes;
uint64_t lost_bytes;
} walk_dbi_t;
struct {
const char *dbi_names[MAX_DBI];
uint64_t dbi_pages[MAX_DBI];
uint64_t dbi_empty_pages[MAX_DBI];
uint64_t dbi_payload_bytes[MAX_DBI];
uint64_t dbi_lost_bytes[MAX_DBI];
walk_dbi_t dbi[MAX_DBI];
short *pagemap;
uint64_t total_payload_bytes;
uint64_t pgcount;
@@ -95,7 +101,7 @@ struct problem *problems_list;
uint64_t total_problems;
static void
#ifdef __GNU__
#ifdef __GNUC__
__attribute__((format(printf, 1, 2)))
#endif
print(const char *msg, ...) {
@@ -110,7 +116,7 @@ static void
}
static void
#ifdef __GNU__
#ifdef __GNUC__
__attribute__((format(printf, 1, 2)))
#endif
error(const char *msg, ...) {
@@ -131,9 +137,9 @@ static void pagemap_cleanup(void) {
int i;
for (i = 1; i < MAX_DBI; ++i) {
if (walk.dbi_names[i]) {
free((void *)walk.dbi_names[i]);
walk.dbi_names[i] = NULL;
if (walk.dbi[i].name) {
free((void *)walk.dbi[i].name);
walk.dbi[i].name = NULL;
}
}
@@ -141,32 +147,35 @@ static void pagemap_cleanup(void) {
walk.pagemap = NULL;
}
static int pagemap_lookup_dbi(const char *dbi) {
static int last;
int i;
static walk_dbi_t *pagemap_lookup_dbi(const char *dbi_name) {
static walk_dbi_t *last;
if (last > 0 && strcmp(walk.dbi_names[last], dbi) == 0)
if (last && strcmp(last->name, dbi_name) == 0)
return last;
for (i = 1; walk.dbi_names[i] && last < MAX_DBI; ++i)
if (strcmp(walk.dbi_names[i], dbi) == 0)
return last = i;
if (i == MAX_DBI)
return -1;
walk.dbi_names[i] = strdup(dbi);
walk_dbi_t *dbi = walk.dbi + 1;
while (dbi->name) {
if (strcmp(dbi->name, dbi_name) == 0)
return last = dbi;
if (++dbi == walk.dbi + MAX_DBI)
return NULL;
}
dbi->name = strdup(dbi_name);
if (verbose > 1) {
print(" - found '%s' area\n", dbi);
print(" - found '%s' area\n", dbi_name);
fflush(NULL);
}
return last = i;
return last = dbi;
}
static void problem_add(const char *object, uint64_t entry_number,
const char *msg, const char *extra, ...) {
static void
#ifdef __GNUC__
__attribute__((format(printf, 4, 5)))
#endif
problem_add(const char *object, uint64_t entry_number, const char *msg,
const char *extra, ...) {
total_problems++;
if (!quiet) {
@@ -233,7 +242,7 @@ static uint64_t problems_pop(struct problem *list) {
}
static int pgvisitor(uint64_t pgno, unsigned pgnumber, void *ctx,
const char *dbi, const char *type, size_t nentries,
const char *dbi_name, const char *type, size_t nentries,
size_t payload_bytes, size_t header_bytes,
size_t unused_bytes) {
(void)ctx;
@@ -241,54 +250,58 @@ static int pgvisitor(uint64_t pgno, unsigned pgnumber, void *ctx,
if (type) {
uint64_t page_bytes = payload_bytes + header_bytes + unused_bytes;
size_t page_size = (size_t)pgnumber * envstat.ms_psize;
int index = pagemap_lookup_dbi(dbi);
if (index < 0)
walk_dbi_t *dbi = pagemap_lookup_dbi(dbi_name);
if (!dbi)
return MDBX_ENOMEM;
if (verbose > 2 && (!only_subdb || strcmp(only_subdb, dbi) == 0)) {
if (verbose > 2 && (!only_subdb || strcmp(only_subdb, dbi_name) == 0)) {
if (pgnumber == 1)
print(" %s-page %" PRIu64, type, pgno);
else
print(" %s-span %" PRIu64 "[%u]", type, pgno, pgnumber);
print(" of %s: header %" PRIiPTR ", payload %" PRIiPTR
", unused %" PRIiPTR "\n",
dbi, header_bytes, payload_bytes, unused_bytes);
dbi_name, header_bytes, payload_bytes, unused_bytes);
}
walk.pgcount += pgnumber;
if (unused_bytes > page_size)
problem_add("page", pgno, "illegal unused-bytes", "%u < %i < %u", 0,
unused_bytes, envstat.ms_psize);
problem_add("page", pgno, "illegal unused-bytes",
"%u < %" PRIuPTR " < %u", 0, unused_bytes, envstat.ms_psize);
if (header_bytes < (int)sizeof(long) ||
(size_t)header_bytes >= envstat.ms_psize - sizeof(long))
problem_add("page", pgno, "illegal header-length",
"%" PRIuPTR " < %i < %" PRIuPTR "", sizeof(long),
"%" PRIuPTR " < %" PRIuPTR " < %" PRIuPTR, sizeof(long),
header_bytes, envstat.ms_psize - sizeof(long));
if (payload_bytes < 1) {
if (nentries > 1) {
problem_add("page", pgno, "zero size-of-entry",
"payload %i bytes, %i entries", payload_bytes, nentries);
"payload %" PRIuPTR " bytes, %" PRIuPTR " entries",
payload_bytes, nentries);
if ((size_t)header_bytes + unused_bytes < page_size) {
/* LY: hush a misuse error */
page_bytes = page_size;
}
} else {
problem_add("page", pgno, "empty", "payload %i bytes, %i entries",
problem_add("page", pgno, "empty",
"payload %" PRIuPTR " bytes, %" PRIuPTR " entries",
payload_bytes, nentries);
walk.dbi_empty_pages[index] += 1;
dbi->pages.empty += 1;
}
}
if (page_bytes != page_size) {
problem_add("page", pgno, "misused",
"%" PRIu64 " != %" PRIu64 " (%ih + %ip + %iu)", page_size,
page_bytes, header_bytes, payload_bytes, unused_bytes);
"%" PRIu64 " != %" PRIu64 " (%" PRIuPTR "h + %" PRIuPTR
"p + %" PRIuPTR "u)",
page_size, page_bytes, header_bytes, payload_bytes,
unused_bytes);
if (page_size > page_bytes)
walk.dbi_lost_bytes[index] += page_size - page_bytes;
dbi->lost_bytes += page_size - page_bytes;
} else {
walk.dbi_payload_bytes[index] += payload_bytes + header_bytes;
dbi->payload_bytes += payload_bytes + header_bytes;
walk.total_payload_bytes += payload_bytes + header_bytes;
}
@@ -299,10 +312,10 @@ static int pgvisitor(uint64_t pgno, unsigned pgnumber, void *ctx,
"%" PRIu64 " > %" PRIu64 "", pgno, lastpgno);
else if (walk.pagemap[pgno])
problem_add("page", pgno, "already used", "in %s",
walk.dbi_names[walk.pagemap[pgno]]);
walk.dbi[walk.pagemap[pgno]].name);
else {
walk.pagemap[pgno] = (short)index;
walk.dbi_pages[index] += 1;
walk.pagemap[pgno] = (short)(dbi - walk.dbi);
dbi->pages.total += 1;
}
++pgno;
} while (--pgnumber);
@@ -337,16 +350,22 @@ static int handle_freedb(const uint64_t record_number, const MDBX_val *key,
problem_add("entry", record_number, "wrong txn-id", "%" PRIaTXN "", txnid);
if (data->iov_len < sizeof(pgno_t) || data->iov_len % sizeof(pgno_t))
problem_add("entry", record_number, "wrong idl size", "%" PRIuPTR "",
problem_add("entry", txnid, "wrong idl size", "%" PRIuPTR "",
data->iov_len);
else {
const pgno_t number = *iptr++;
if (number >= MDBX_PNL_UM_MAX)
problem_add("entry", record_number, "wrong idl length", "%" PRIiPTR "",
number);
else if ((number + 1) * sizeof(pgno_t) != data->iov_len)
problem_add("entry", record_number, "mismatch idl length",
"%" PRIuSIZE " != %" PRIuSIZE "",
if (number < 1 || number >= INT_MAX / 2)
problem_add("entry", txnid, "wrong idl length", "%" PRIaPGNO, number);
else if ((number + 1) * sizeof(pgno_t) > data->iov_len)
problem_add("entry", txnid, "trimmed idl",
"%" PRIuSIZE " > %" PRIuSIZE " (corruption)",
(number + 1) * sizeof(pgno_t), data->iov_len);
else if (data->iov_len - (number + 1) * sizeof(pgno_t) >=
/* LY: allow gap upto one page. it is ok
* and better than shink-and-retry inside mdbx_update_gc() */
envstat.ms_psize)
problem_add("entry", txnid, "extra idl space",
"%" PRIuSIZE " < %" PRIuSIZE " (minor, not a trouble)",
(number + 1) * sizeof(pgno_t), data->iov_len);
else {
freedb_pages += number;
@@ -359,12 +378,12 @@ static int handle_freedb(const uint64_t record_number, const MDBX_val *key,
for (unsigned i = 0; i < number; ++i) {
const pgno_t pg = iptr[i];
if (pg < NUM_METAS || pg > envinfo.mi_last_pgno)
problem_add("entry", record_number, "wrong idl entry",
problem_add("entry", txnid, "wrong idl entry",
"%u < %" PRIaPGNO " < %" PRIu64 "", NUM_METAS, pg,
envinfo.mi_last_pgno);
else if (MDBX_PNL_DISORDERED(prev, pg)) {
bad = " [bad sequence]";
problem_add("entry", record_number, "bad sequence",
problem_add("entry", txnid, "bad sequence",
"%" PRIaPGNO " <> %" PRIaPGNO "", prev, pg);
}
prev = pg;
@@ -516,7 +535,7 @@ static int process_db(MDBX_dbi dbi, char *name, visitor *handler, bool silent) {
if (key.iov_len > maxkeysize) {
problem_add("entry", record_count, "key length exceeds max-key-size",
"%" PRIuPTR " > %u", key.iov_len, maxkeysize);
"%" PRIuPTR " > %" PRIuPTR, key.iov_len, maxkeysize);
} else if ((flags & MDBX_INTEGERKEY) && key.iov_len != sizeof(uint64_t) &&
key.iov_len != sizeof(uint32_t)) {
problem_add("entry", record_count, "wrong key length",
@@ -758,7 +777,7 @@ static void print_size(const char *prefix, const uint64_t value,
}
int main(int argc, char *argv[]) {
int i, rc;
int rc;
char *prog = argv[0];
char *envname;
int problems_maindb = 0, problems_freedb = 0, problems_meta = 0;
@@ -778,14 +797,14 @@ int main(int argc, char *argv[]) {
}
#endif
walk.dbi_names[0] = "@gc";
walk.dbi[FREE_DBI].name = "@gc";
atexit(pagemap_cleanup);
if (argc < 2) {
usage(prog);
}
while ((i = getopt(argc, argv, "Vvqnwcds:")) != EOF) {
for (int i; (i = getopt(argc, argv, "Vvqnwcds:")) != EOF;) {
switch (i) {
case 'V':
printf("%s (%s, build %s)\n", mdbx_version.git.describe,
@@ -837,8 +856,8 @@ int main(int argc, char *argv[]) {
#endif /* !WINDOWS */
envname = argv[optind];
print("Running mdbx_chk for 'read-%s' in %s mode...\n", envname,
(envflags & MDBX_RDONLY) ? "read" : "write");
print("Running mdbx_chk for %s in 'read-%s' mode...\n", envname,
(envflags & MDBX_RDONLY) ? "only" : "write");
fflush(NULL);
rc = mdbx_env_create(&env);
@@ -988,24 +1007,25 @@ int main(int argc, char *argv[]) {
goto bailout;
}
uint64_t n;
for (n = 0; n < lastpgno; ++n)
for (uint64_t n = 0; n < lastpgno; ++n)
if (!walk.pagemap[n])
walk.dbi_pages[0] += 1;
walk.dbi[FREE_DBI].pages.total += 1;
empty_pages = lost_bytes = 0;
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
empty_pages += walk.dbi_empty_pages[i];
lost_bytes += walk.dbi_lost_bytes[i];
for (walk_dbi_t *dbi = walk.dbi; ++dbi < walk.dbi + MAX_DBI && dbi->name;) {
empty_pages += dbi->pages.empty;
lost_bytes += dbi->lost_bytes;
}
if (verbose) {
uint64_t total_page_bytes = walk.pgcount * envstat.ms_psize;
print(" - dbi pages: %" PRIu64 " total", walk.pgcount);
if (verbose > 1)
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i)
print(", %s %" PRIu64 "", walk.dbi_names[i], walk.dbi_pages[i]);
print(", %s %" PRIu64 "\n", walk.dbi_names[0], walk.dbi_pages[0]);
for (walk_dbi_t *dbi = walk.dbi;
++dbi < walk.dbi + MAX_DBI && dbi->name;)
print(", %s %" PRIu64, dbi->name, dbi->pages.total);
print(", %s %" PRIu64 "\n", walk.dbi[FREE_DBI].name,
walk.dbi[FREE_DBI].pages.total);
if (verbose > 1) {
print(" - space info: total %" PRIu64 " bytes, payload %" PRIu64
" (%.1f%%), unused "
@@ -1015,19 +1035,19 @@ int main(int argc, char *argv[]) {
total_page_bytes - walk.total_payload_bytes,
(total_page_bytes - walk.total_payload_bytes) * 100.0 /
total_page_bytes);
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
uint64_t dbi_bytes = walk.dbi_pages[i] * envstat.ms_psize;
for (walk_dbi_t *dbi = walk.dbi;
++dbi < walk.dbi + MAX_DBI && dbi->name;) {
uint64_t dbi_bytes = dbi->pages.total * envstat.ms_psize;
print(" %s: subtotal %" PRIu64 " bytes (%.1f%%),"
" payload %" PRIu64 " (%.1f%%), unused %" PRIu64 " (%.1f%%)",
walk.dbi_names[i], dbi_bytes,
dbi_bytes * 100.0 / total_page_bytes, walk.dbi_payload_bytes[i],
walk.dbi_payload_bytes[i] * 100.0 / dbi_bytes,
dbi_bytes - walk.dbi_payload_bytes[i],
(dbi_bytes - walk.dbi_payload_bytes[i]) * 100.0 / dbi_bytes);
if (walk.dbi_empty_pages[i])
print(", %" PRIu64 " empty pages", walk.dbi_empty_pages[i]);
if (walk.dbi_lost_bytes[i])
print(", %" PRIu64 " bytes lost", walk.dbi_lost_bytes[i]);
dbi->name, dbi_bytes, dbi_bytes * 100.0 / total_page_bytes,
dbi->payload_bytes, dbi->payload_bytes * 100.0 / dbi_bytes,
dbi_bytes - dbi->payload_bytes,
(dbi_bytes - dbi->payload_bytes) * 100.0 / dbi_bytes);
if (dbi->pages.empty)
print(", %" PRIu64 " empty pages", dbi->pages.empty);
if (dbi->lost_bytes)
print(", %" PRIu64 " bytes lost", dbi->lost_bytes);
print("\n");
}
}
@@ -1084,9 +1104,9 @@ int main(int argc, char *argv[]) {
error("used pages mismatch (%" PRIu64 " != %" PRIu64 ")\n",
walk.pgcount, lastpgno - freedb_pages);
}
if (walk.dbi_pages[0] != freedb_pages) {
if (walk.dbi[FREE_DBI].pages.total != freedb_pages) {
error("gc pages mismatch (%" PRIu64 " != %" PRIu64 ")\n",
walk.dbi_pages[0], freedb_pages);
walk.dbi[FREE_DBI].pages.total, freedb_pages);
}
} else if (verbose) {
print(" - skip check used and gc pages (btree-traversal with "

View File

@@ -143,8 +143,8 @@ static void readhdr(void) {
lineno, (char *)dbuf.iov_base + STRLENOF("mapsize="));
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.iov_base, "maxreaders=",
STRLENOF("maxreaders="))) {
} else if (!strncmp(dbuf.iov_base,
"maxreaders=", STRLENOF("maxreaders="))) {
int i;
ptr = memchr(dbuf.iov_base, '\n', dbuf.iov_len);
if (ptr)

View File

@@ -152,9 +152,15 @@ int main(int argc, char *argv[]) {
goto env_close;
}
if (envinfo || freinfo) {
(void)mdbx_env_info(env, &mei, sizeof(mei));
} else {
/* LY: zap warnings from gcc */
memset(&mei, 0, sizeof(mei));
}
if (envinfo) {
(void)mdbx_env_stat(env, &mst, sizeof(mst));
(void)mdbx_env_info(env, &mei, sizeof(mei));
printf("Environment Info\n");
printf(" Pagesize: %u\n", mst.ms_psize);
if (mei.mi_geo.lower != mei.mi_geo.upper) {
@@ -183,7 +189,6 @@ int main(int argc, char *argv[]) {
} else {
/* LY: zap warnings from gcc */
memset(&mst, 0, sizeof(mst));
memset(&mei, 0, sizeof(mei));
}
if (rdrinfo) {

View File

@@ -18,8 +18,8 @@
#error "API version mismatch!"
#endif
#define MDBX_VERSION_RELEASE 0
#define MDBX_VERSION_REVISION 0
#define MDBX_VERSION_RELEASE 6
#define MDBX_VERSION_REVISION 1
/*LIBMDBX_EXPORTS*/ const mdbx_version_info mdbx_version = {
MDBX_VERSION_MAJOR,

View File

@@ -32,10 +32,10 @@
#endif /* _MSC_VER (warnings) */
/* If you wish to build your application for a previous Windows platform,
* include WinSDKVer.h and set the _WIN32_WINNT macro to the platform you
* wish to support before including SDKDDKVer.h.
*
* TODO: #define _WIN32_WINNT WIN32_MUSTDIE */
* include WinSDKVer.h and set the _WIN32_WINNT macro to the platform you
* wish to support before including SDKDDKVer.h.
*
* TODO: #define _WIN32_WINNT WIN32_MUSTDIE */
#include <SDKDDKVer.h>
#endif /* WINDOWS */

View File

@@ -43,6 +43,11 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
if (narg + 1 < argc && strncmp("--", argv[narg + 1], 2) != 0) {
*value = argv[narg + 1];
if (strcmp(*value, "default") == 0) {
if (!default_value)
failure("Option '--%s' doen't accept default value\n", option);
*value = default_value;
}
++narg;
return true;
}
@@ -57,9 +62,15 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty) {
return parse_option(argc, argv, narg, option, value, allow_empty,
allow_empty ? "" : nullptr);
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty,
const char *default_value) {
const char *value_cstr;
if (!parse_option(argc, argv, narg, option, &value_cstr,
allow_empty ? "" : nullptr))
if (!parse_option(argc, argv, narg, option, &value_cstr, default_value))
return false;
if (!allow_empty && strlen(value_cstr) == 0)
@@ -75,7 +86,7 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
if (!parse_option(argc, argv, narg, option, &list))
return false;
mask = 0;
unsigned clear = 0;
while (*list) {
if (*list == ',' || *list == ' ' || *list == '\t') {
++list;
@@ -83,14 +94,21 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
}
const char *const comma = strchr(list, ',');
const bool strikethrough = *list == '-' || *list == '~';
if (strikethrough || *list == '+')
++list;
else
mask = clear;
const size_t len = (comma) ? comma - list : strlen(list);
const option_verb *scan = verbs;
while (true) {
if (!scan->verb)
failure("Unknown verb '%.*s', for option '==%s'\n", (int)len, list,
option);
if (strlen(scan->verb) == len && strncmp(list, scan->verb, len) == 0) {
mask |= scan->mask;
mask = strikethrough ? mask & ~scan->mask : mask | scan->mask;
clear = strikethrough ? clear & ~scan->mask : clear | scan->mask;
list += len;
break;
}
@@ -103,15 +121,36 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint64_t &value, const scale_mode scale,
const uint64_t minval, const uint64_t maxval) {
const uint64_t minval, const uint64_t maxval,
const uint64_t default_value) {
const char *value_cstr;
if (!parse_option(argc, argv, narg, option, &value_cstr))
return false;
if (default_value && strcmp(value_cstr, "default") == 0) {
value = default_value;
return true;
}
if (strcmp(value_cstr, "min") == 0 || strcmp(value_cstr, "minimal") == 0) {
value = minval;
return true;
}
if (strcmp(value_cstr, "max") == 0 || strcmp(value_cstr, "maximal") == 0) {
value = maxval;
return true;
}
char *suffix = nullptr;
errno = 0;
unsigned long raw = strtoul(value_cstr, &suffix, 0);
unsigned long long raw = strtoull(value_cstr, &suffix, 0);
if ((suffix && *suffix) || errno) {
suffix = nullptr;
errno = 0;
raw = strtoull(value_cstr, &suffix, 10);
}
if (errno)
failure("Option '--%s' expects a numeric value (%s)\n", option,
test_strerror(errno));
@@ -167,28 +206,58 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
unsigned &value, const scale_mode scale,
const unsigned minval, const unsigned maxval) {
const unsigned minval, const unsigned maxval,
const unsigned default_value) {
uint64_t huge;
if (!parse_option(argc, argv, narg, option, huge, scale, minval, maxval))
if (!parse_option(argc, argv, narg, option, huge, scale, minval, maxval,
default_value))
return false;
value = (unsigned)huge;
return true;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint8_t &value, const uint8_t minval, const uint8_t maxval) {
uint8_t &value, const uint8_t minval, const uint8_t maxval,
const uint8_t default_value) {
uint64_t huge;
if (!parse_option(argc, argv, narg, option, huge, no_scale, minval, maxval))
if (!parse_option(argc, argv, narg, option, huge, no_scale, minval, maxval,
default_value))
return false;
value = (uint8_t)huge;
return true;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int64_t &value, const int64_t minval, const int64_t maxval,
const int64_t default_value) {
uint64_t proxy = (uint64_t)value;
if (parse_option(argc, argv, narg, option, proxy, config::binary,
(uint64_t)minval, (uint64_t)maxval,
(uint64_t)default_value)) {
value = (int64_t)proxy;
return true;
}
return false;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int32_t &value, const int32_t minval, const int32_t maxval,
const int32_t default_value) {
uint64_t proxy = (uint64_t)value;
if (parse_option(argc, argv, narg, option, proxy, config::binary,
(uint64_t)minval, (uint64_t)maxval,
(uint64_t)default_value)) {
value = (int32_t)proxy;
return true;
}
return false;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool &value) {
const char *value_cstr = NULL;
const char *value_cstr = nullptr;
if (!parse_option(argc, argv, narg, option, &value_cstr, "yes")) {
const char *current = argv[narg];
if (strncmp(current, "--no-", 5) == 0 && strcmp(current + 5, option) == 0) {
@@ -257,7 +326,7 @@ static void dump_verbs(const char *caption, size_t bits,
++verbs;
}
logging::feed("\n");
logging::feed("%s\n", (*comma == '\0') ? "none" : "");
}
static void dump_duration(const char *caption, unsigned duration) {
@@ -288,8 +357,12 @@ void dump(const char *title) {
: i->params.pathname_log.c_str());
}
log_info("database: %s, size %" PRIu64 "\n", i->params.pathname_db.c_str(),
i->params.size);
log_info("database: %s, size %" PRIuPTR "[%" PRIiPTR "..%" PRIiPTR
", %i %i, %i]\n",
i->params.pathname_db.c_str(), i->params.size_now,
i->params.size_lower, i->params.size_upper,
i->params.shrink_threshold, i->params.growth_step,
i->params.pagesize);
dump_verbs("mode", i->params.mode_flags, mode_bits);
dump_verbs("table", i->params.table_flags, table_bits);
@@ -306,7 +379,13 @@ void dump(const char *title) {
log_info("threads %u\n", i->params.nthreads);
log_info("keygen.case: %s\n", keygencase2str(i->params.keygen.keycase));
log_info(
"keygen.params: case %s, width %u, mesh %u, rotate %u, offset %" PRIu64
", split %u/%u\n",
keygencase2str(i->params.keygen.keycase), i->params.keygen.width,
i->params.keygen.mesh, i->params.keygen.rotate, i->params.keygen.offset,
i->params.keygen.split,
i->params.keygen.width - i->params.keygen.split);
log_info("keygen.seed: %u\n", i->params.keygen.seed);
log_info("key: minlen %u, maxlen %u\n", i->params.keylen_min,
i->params.keylen_max);
@@ -469,3 +548,27 @@ bool actor_config::deserialize(const char *str, actor_config &config) {
TRACE("<< actor_config::deserialize: OK\n");
return true;
}
unsigned actor_params::mdbx_keylen_min() const {
return (table_flags & MDBX_INTEGERKEY) ? 4 : 0;
}
unsigned actor_params::mdbx_keylen_max() const {
return (table_flags & MDBX_INTEGERKEY)
? 8
: std::min((unsigned)mdbx_get_maxkeysize(pagesize),
(unsigned)UINT16_MAX);
}
unsigned actor_params::mdbx_datalen_min() const {
return (table_flags & MDBX_INTEGERDUP) ? 4 : 0;
}
unsigned actor_params::mdbx_datalen_max() const {
return (table_flags & MDBX_INTEGERDUP)
? 8
: std::min((table_flags & MDBX_DUPSORT)
? (unsigned)mdbx_get_maxkeysize(pagesize)
: (unsigned)MDBX_MAXDATASIZE,
(unsigned)UINT16_MAX);
}

View File

@@ -62,6 +62,10 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty = false);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty,
const char *default_value);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool &value);
@@ -75,16 +79,25 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint64_t &value, const scale_mode scale,
const uint64_t minval = 0, const uint64_t maxval = INT64_MAX);
const uint64_t minval = 0, const uint64_t maxval = INT64_MAX,
const uint64_t default_value = 0);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
unsigned &value, const scale_mode scale,
const unsigned minval = 0, const unsigned maxval = INT32_MAX);
const unsigned minval = 0, const unsigned maxval = INT32_MAX,
const unsigned default_value = 0);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint8_t &value, const uint8_t minval = 0,
const uint8_t maxval = 255);
const uint8_t maxval = 255, const uint8_t default_value = 0);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int64_t &value, const int64_t minval, const int64_t maxval,
const int64_t default_value = -1);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int32_t &value, const int32_t minval, const int32_t maxval,
const int32_t default_value = -1);
//-----------------------------------------------------------------------------
#pragma pack(push, 1)
@@ -121,6 +134,8 @@ struct keygen_params_pod {
* Иначе говоря, нет смысла в со-координации генерации паттернов для
* ключей и значений. Более того, генерацию значений всегда необходимо
* рассматривать в контексте связки с одним значением ключа.
* - Тем не менее, во всех случаях достаточно важным является равномерная
* всех возможных сочетаний длин ключей и данных.
*
* width:
* Большинство тестов предполагают создание или итерирование некоторого
@@ -156,7 +171,7 @@ struct keygen_params_pod {
* псевдо-случайные значений ключей без псевдо-случайности в значениях.
*
* Такое ограничение соответствуют внутренней алгоритмике libmdbx. Проще
* говоря мы можем проверить движок псевдо-случайной последовательностью
* говоря, мы можем проверить движок псевдо-случайной последовательностью
* ключей на таблицах без дубликатов (без multi-value), а затем проверить
* корректность работу псевдо-случайной последовательностью значений на
* таблицах с дубликатами (с multi-value), опционально добавляя
@@ -203,7 +218,12 @@ struct actor_params_pod {
unsigned mode_flags;
unsigned table_flags;
uint64_t size;
intptr_t size_lower;
intptr_t size_now;
intptr_t size_upper;
int shrink_threshold;
int growth_step;
int pagesize;
unsigned test_duration;
unsigned test_nops;
@@ -246,6 +266,11 @@ struct actor_params : public config::actor_params_pod {
std::string pathname_log;
std::string pathname_db;
void set_defaults(const std::string &tmpdir);
unsigned mdbx_keylen_min() const;
unsigned mdbx_keylen_max() const;
unsigned mdbx_datalen_min() const;
unsigned mdbx_datalen_max() const;
};
struct actor_config : public config::actor_config_pod {

65
test/gc.sh Executable file
View File

@@ -0,0 +1,65 @@
#!/bin/bash
set -euo pipefail
make check
TESTDB_PREFIX=${1:-/dev/shm/mdbx-gc-test}.
function rep9 { printf "%*s" $1 '' | tr ' ' '9'; }
function join { local IFS="$1"; shift; echo "$*"; }
function bit2option { local -n arr=$1; (( ($2&(1<<$3)) != 0 )) && echo -n '+' || echo -n '-'; echo "${arr[$3]}"; }
options=(writemap coalesce lifo)
function bits2list {
local -n arr=$1
local i
local list=()
for ((i=0; i<${#arr[@]}; ++i)) do
list[$i]=$(bit2option $1 $2 $i)
done
join , "${list[@]}"
}
function probe {
echo "=============================================== $(date)"
echo "${caption}: $*"
rm -f ${TESTDB_PREFIX}* \
&& ./mdbx_test --pathname=${TESTDB_PREFIX}db "$@" | lz4 > ${TESTDB_PREFIX}log.lz4 \
&& ./mdbx_chk -nvv ${TESTDB_PREFIX}db | tee ${TESTDB_PREFIX}chk \
|| (echo "FAILED"; exit 1)
}
###############################################################################
count=0
for nops in {2..7}; do
for ((wbatch=nops-1; wbatch > 0; --wbatch)); do
loops=$(((3333 >> nops) / nops + 1))
for ((rep=0; rep++ < loops; )); do
for ((bits=2**${#options[@]}; --bits >= 0; )); do
seed=$(date +%N)
caption="Probe #$((++count)) int-key,w/o-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+key.integer,-data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) int-key,with-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+key.integer,+data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) int-key,int-data, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+key.integer,+data.integer --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) w/o-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=-data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} --hill
caption="Probe #$((++count)) with-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} --hill
done
done
done
done
echo "=== ALL DONE ====================== $(date)"

View File

@@ -53,7 +53,7 @@ bool testcase_hill::run() {
*/
/* TODO: работа в несколько потоков */
keyvalue_maker.setup(config.params, 0 /* thread_number */);
keyvalue_maker.setup(config.params, config.actor_id, 0 /* thread_number */);
keygen::buffer a_key = keygen::alloc(config.params.keylen_max);
keygen::buffer a_data_0 = keygen::alloc(config.params.datalen_max);
@@ -65,7 +65,9 @@ bool testcase_hill::run() {
? MDBX_NODUPDATA
: MDBX_NODUPDATA | MDBX_NOOVERWRITE;
const unsigned update_flags =
MDBX_CURRENT | MDBX_NODUPDATA | MDBX_NOOVERWRITE;
(config.params.table_flags & MDBX_DUPSORT)
? MDBX_CURRENT | MDBX_NODUPDATA | MDBX_NOOVERWRITE
: MDBX_NODUPDATA;
uint64_t serial_count = 0;
unsigned txn_nops = 0;
@@ -115,7 +117,7 @@ bool testcase_hill::run() {
rc = mdbx_replace(txn_guard.get(), dbi, &a_key->value, &a_data_0->value,
&a_data_1->value, update_flags);
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_put(update-a: 1->0)", rc);
failure_perror("mdbx_replace(update-a: 1->0)", rc);
if (++txn_nops >= config.params.batch_write) {
txn_restart(false, false);
@@ -156,8 +158,6 @@ bool testcase_hill::run() {
a_serial);
generate_pair(a_serial, a_key, a_data_0, 0);
generate_pair(a_serial, a_key, a_data_1, age_shift);
if (a_serial == 808)
log_trace("!!!");
int rc = mdbx_replace(txn_guard.get(), dbi, &a_key->value, &a_data_1->value,
&a_data_0->value, update_flags);
if (unlikely(rc != MDBX_SUCCESS))

View File

@@ -58,7 +58,11 @@ bool testcase_jitter::run() {
jitter_delay();
db_close();
report(1);
/* just 'align' nops with other tests with batching */
const auto batching =
std::max(config.params.batch_read, config.params.batch_write);
report(std::max(1u, batching / 2));
}
return true;
}

View File

@@ -30,7 +30,7 @@ serial_t injective(const serial_t serial,
/* LY: All these "magic" prime numbers were found
* and verified with a bit of brute force. */
static const uint64_t m[64 - serial_minwith] = {
static const uint64_t m[64 - serial_minwith + 1] = {
/* 8 - 24 */
113, 157, 397, 653, 1753, 5641, 9697, 23873, 25693, 80833, 105953, 316937,
309277, 834497, 1499933, 4373441, 10184137,
@@ -43,26 +43,31 @@ serial_t injective(const serial_t serial,
2420886491930041, 3601632139991929, 11984491914483833, 21805846439714153,
23171543400565993, 53353226456762893, 155627817337932409,
227827205384840249, 816509268558278821, 576933057762605689,
2623957345935638441, 5048241705479929949, 4634245581946485653};
static const uint8_t s[64 - serial_minwith] = {
2623957345935638441, 5048241705479929949, 4634245581946485653,
4613509448041658233, 4952535426879925961};
static const uint8_t s[64 - serial_minwith + 1] = {
/* 8 - 24 */
2, 3, 4, 4, 2, 4, 3, 3, 7, 3, 3, 4, 8, 3, 10, 3, 11,
/* 25 - 64 */
11, 9, 9, 9, 11, 10, 5, 14, 11, 16, 14, 12, 13, 16, 19, 10, 10, 21, 7, 20,
10, 14, 22, 19, 3, 21, 18, 19, 26, 24, 2, 21, 25, 29, 24, 10, 11, 14};
10, 14, 22, 19, 3, 21, 18, 19, 26, 24, 2, 21, 25, 29, 24, 10, 11, 14, 20,
19};
serial_t result = serial * m[bits - 8];
const auto mult = m[bits - 8];
const auto shift = s[bits - 8];
serial_t result = serial * mult;
if (salt) {
const unsigned left = bits / 2;
const unsigned right = bits - left;
result = (result << left) | ((result & mask(bits)) >> right);
result = (result ^ salt) * m[bits - 8];
result = (result ^ salt) * mult;
}
result ^= result << s[bits - 8];
result ^= result << shift;
result &= mask(bits);
log_trace("keygen-injective: serial %" PRIu64 " into %" PRIu64, serial,
result);
log_trace("keygen-injective: serial %" PRIu64 "/%u @%" PRIx64 ",%u,%" PRIu64
" => %" PRIu64 "/%u",
serial, bits, mult, shift, salt, result, bits);
return result;
}
@@ -73,8 +78,9 @@ void __hot maker::pair(serial_t serial, const buffer &key, buffer &value,
assert(mapping.mesh <= mapping.width);
assert(mapping.rotate <= mapping.width);
assert(mapping.offset <= mask(mapping.width));
assert(!(key_essentials.flags & (MDBX_INTEGERDUP | MDBX_REVERSEDUP)));
assert(!(value_essentials.flags & (MDBX_INTEGERKEY | MDBX_REVERSEKEY)));
assert(!(key_essentials.flags &
~(MDBX_INTEGERKEY | MDBX_REVERSEKEY | MDBX_DUPSORT)));
assert(!(value_essentials.flags & ~(MDBX_INTEGERDUP | MDBX_REVERSEDUP)));
log_trace("keygen-pair: serial %" PRIu64 ", data-age %" PRIu64, serial,
value_age);
@@ -82,31 +88,49 @@ void __hot maker::pair(serial_t serial, const buffer &key, buffer &value,
if (mapping.mesh >= serial_minwith) {
serial =
(serial & ~mask(mapping.mesh)) | injective(serial, mapping.mesh, salt);
log_trace("keygen-pair: mesh %" PRIu64, serial);
log_trace("keygen-pair: mesh@%u => %" PRIu64, mapping.mesh, serial);
}
if (mapping.rotate) {
const unsigned right = mapping.rotate;
const unsigned left = mapping.width - right;
serial = (serial << left) | ((serial & mask(mapping.width)) >> right);
log_trace("keygen-pair: rotate %" PRIu64 ", 0x%" PRIx64, serial, serial);
log_trace("keygen-pair: rotate@%u => %" PRIu64 ", 0x%" PRIx64,
mapping.rotate, serial, serial);
}
serial = (serial + mapping.offset) & mask(mapping.width);
log_trace("keygen-pair: offset %" PRIu64, serial);
serial += base;
if (mapping.offset) {
serial = (serial + mapping.offset) & mask(mapping.width);
log_trace("keygen-pair: offset@%" PRIu64 " => %" PRIu64, mapping.offset,
serial);
}
if (base) {
serial += base;
log_trace("keygen-pair: base@%" PRIu64 " => %" PRIu64, base, serial);
}
serial_t key_serial = serial;
serial_t value_serial = value_age;
serial_t value_serial = value_age << mapping.split;
if (mapping.split) {
key_serial = serial >> mapping.split;
value_serial =
(serial & mask(mapping.split)) | (value_age << mapping.split);
if (key_essentials.flags & MDBX_DUPSORT) {
key_serial >>= mapping.split;
value_serial += serial & mask(mapping.split);
} else {
/* Без MDBX_DUPSORT требуется уникальность ключей, а для этого нельзя
* отбрасывать какие-либо биты serial после инъективного преобразования.
* Поэтому key_serial не трогаем, а в value_serial нелинейно вмешиваем
* запрошенное количество бит из serial */
value_serial +=
(serial ^ (serial >> mapping.split)) & mask(mapping.split);
}
value_serial |= value_age << mapping.split;
log_trace("keygen-pair: split@%u => k%" PRIu64 ", v%" PRIu64, mapping.split,
key_serial, value_serial);
}
log_trace("keygen-pair: key %" PRIu64 ", value %" PRIu64, key_serial,
value_serial);
mk(key_serial, key_essentials, *key);
mk(value_serial, value_essentials, *value);
@@ -118,26 +142,26 @@ void __hot maker::pair(serial_t serial, const buffer &key, buffer &value,
}
}
void maker::setup(const config::actor_params_pod &actor,
void maker::setup(const config::actor_params_pod &actor, unsigned actor_id,
unsigned thread_number) {
key_essentials.flags =
actor.table_flags & (MDBX_INTEGERKEY | MDBX_REVERSEKEY);
assert(actor.keylen_min < UINT8_MAX);
actor.table_flags & (MDBX_INTEGERKEY | MDBX_REVERSEKEY | MDBX_DUPSORT);
assert(actor.keylen_min <= UINT8_MAX);
key_essentials.minlen = (uint8_t)actor.keylen_min;
assert(actor.keylen_max < UINT16_MAX);
assert(actor.keylen_max <= UINT16_MAX);
key_essentials.maxlen = (uint16_t)actor.keylen_max;
value_essentials.flags =
actor.table_flags & (MDBX_INTEGERDUP | MDBX_REVERSEDUP);
assert(actor.datalen_min < UINT8_MAX);
assert(actor.datalen_min <= UINT8_MAX);
value_essentials.minlen = (uint8_t)actor.datalen_min;
assert(actor.datalen_max < UINT16_MAX);
assert(actor.datalen_max <= UINT16_MAX);
value_essentials.maxlen = (uint16_t)actor.datalen_max;
assert(thread_number < 2);
(void)thread_number;
mapping = actor.keygen;
salt = actor.keygen.seed * UINT64_C(14653293970879851569);
salt = (actor.keygen.seed + actor_id) * UINT64_C(14653293970879851569);
// FIXME: TODO
base = 0;
@@ -165,7 +189,7 @@ bool maker::increment(serial_t &serial, int delta) {
//-----------------------------------------------------------------------------
size_t length(serial_t serial) {
static size_t length(serial_t serial) {
size_t n = 0;
if (serial > UINT32_MAX) {
n = 4;
@@ -199,7 +223,10 @@ void __hot maker::mk(const serial_t serial, const essentials &params,
assert(params.maxlen >= length(serial));
out.value.iov_base = out.bytes;
out.value.iov_len = params.minlen;
out.value.iov_len =
(params.maxlen > params.minlen)
? params.minlen + serial % (params.maxlen - params.minlen)
: params.minlen;
if (params.flags & (MDBX_INTEGERKEY | MDBX_INTEGERDUP)) {
assert(params.maxlen == params.minlen);

View File

@@ -44,7 +44,7 @@ namespace keygen {
* - абсолютное значение ключей или разность между отдельными значениями;
*
* Соответственно, в общих чертах, схема генерации следующая:
* - вводится плоская одномерная "координата" uint64_t;
* - вводится плоская одномерная "координата" serial (uint64_t);
* - генерация специфических паттернов (последовательностей)
* реализуется посредством соответствующих преобразований "координат", при
* этом все подобные преобразования выполняются только над "координатой";
@@ -74,7 +74,7 @@ typedef uint64_t serial_t;
enum : serial_t {
serial_minwith = 8,
serial_maxwith = sizeof(serial_t) * 8,
serial_allones = ~(serial_t)0
serial_allones = ~(serial_t)0u
};
struct result {
@@ -85,6 +85,10 @@ struct result {
uint32_t u32;
uint64_t u64;
};
std::string as_string() const {
return std::string((const char *)value.iov_base, value.iov_len);
}
};
//-----------------------------------------------------------------------------
@@ -115,11 +119,10 @@ public:
void pair(serial_t serial, const buffer &key, buffer &value,
serial_t value_age);
void setup(const config::actor_params_pod &actor, unsigned thread_number);
void setup(const config::actor_params_pod &actor, unsigned actor_id,
unsigned thread_number);
bool increment(serial_t &serial, int delta);
};
size_t length(serial_t serial);
} /* namespace keygen */

View File

@@ -37,6 +37,31 @@ void __noreturn failure_perror(const char *what, int errnum) {
//-----------------------------------------------------------------------------
static void mdbx_logger(int type, const char *function, int line,
const char *msg, va_list args) {
logging::loglevel level = logging::info;
if (type & MDBX_DBG_EXTRA)
level = logging::extra;
if (type & MDBX_DBG_TRACE)
level = logging::trace;
if (type & MDBX_DBG_PRINT)
level = logging::verbose;
if (!function)
function = "unknown";
if (type & MDBX_DBG_ASSERT) {
log_error("mdbx: assertion failure: %s, %d", function, line);
level = logging::failure;
}
if (logging::output(
level,
strncmp(function, "mdbx_", 5) == 0 ? "%s: " : "mdbx: %s: ", function))
logging::feed_ap(msg, args);
if (type & MDBX_DBG_ASSERT)
abort();
}
namespace logging {
static std::string prefix;
@@ -44,8 +69,19 @@ static std::string suffix;
static loglevel level;
static FILE *last;
void setup(loglevel _level, const std::string &_prefix) {
void setlevel(loglevel _level) {
level = (_level > error) ? failure : _level;
int mdbx_dbg_opts = MDBX_DBG_ASSERT | MDBX_DBG_JITTER | MDBX_DBG_DUMP;
if (level <= trace)
mdbx_dbg_opts |= MDBX_DBG_TRACE;
if (level <= verbose)
mdbx_dbg_opts |= MDBX_DBG_PRINT;
int rc = mdbx_setup_debug(mdbx_dbg_opts, mdbx_logger);
log_trace("set mdbx debug-opts: 0x%02x", rc);
}
void setup(loglevel _level, const std::string &_prefix) {
setlevel(_level);
prefix = _prefix;
}
@@ -157,7 +193,7 @@ bool output(const logging::loglevel priority, const char *format, va_list ap) {
return true;
}
bool feed(const char *format, va_list ap) {
bool feed_ap(const char *format, va_list ap) {
if (!last)
return false;
@@ -176,7 +212,7 @@ bool feed(const char *format, ...) {
va_list ap;
va_start(ap, format);
feed(format, ap);
feed_ap(format, ap);
va_end(ap);
return true;
}
@@ -206,7 +242,7 @@ void local_suffix::pop() {
local_suffix::~local_suffix() { suffix.erase(trim_pos); }
} /* namespace log */
} // namespace logging
void log_extra(const char *msg, ...) {
if (logging::extra >= logging::level) {

View File

@@ -46,11 +46,12 @@ enum loglevel {
const char *level2str(const loglevel level);
void setup(loglevel level, const std::string &prefix);
void setup(const std::string &prefix);
void setlevel(loglevel level);
bool output(const loglevel priority, const char *format, va_list ap);
bool __printf_args(2, 3)
output(const loglevel priority, const char *format, ...);
bool feed(const char *format, va_list ap);
bool feed_ap(const char *format, va_list ap);
bool __printf_args(1, 2) feed(const char *format, ...);
class local_suffix {
@@ -70,7 +71,7 @@ public:
~local_suffix();
};
} /* namespace log */
} // namespace logging
void __printf_args(1, 2) log_extra(const char *msg, ...);
void __printf_args(1, 2) log_trace(const char *msg, ...);

View File

@@ -1,4 +1,4 @@
/*
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
@@ -35,25 +35,31 @@ void actor_params::set_defaults(const std::string &tmpdir) {
mode_flags = MDBX_NOSUBDIR | MDBX_WRITEMAP | MDBX_MAPASYNC | MDBX_NORDAHEAD |
MDBX_NOMEMINIT | MDBX_COALESCE | MDBX_LIFORECLAIM;
table_flags = MDBX_DUPSORT;
size = 1024 * 1024 * 4;
size_lower = -1;
size_now = 1024 * 1024 * ((table_flags & MDBX_DUPSORT) ? 4 : 256);
size_upper = -1;
shrink_threshold = -1;
growth_step = -1;
pagesize = -1;
keygen.seed = 1;
keygen.keycase = kc_random;
keygen.width = 32;
keygen.mesh = 32;
keygen.width = (table_flags & MDBX_DUPSORT) ? 32 : 64;
keygen.mesh = keygen.width;
keygen.split = keygen.width / 2;
keygen.rotate = 0;
keygen.offset = 0;
keygen.rotate = 3;
keygen.offset = 41;
test_duration = 0;
test_nops = 1000;
nrepeat = 1;
nthreads = 1;
keylen_min = 0;
keylen_max = 42;
datalen_min = 0;
datalen_max = 256;
keylen_min = mdbx_keylen_min();
keylen_max = mdbx_keylen_max();
datalen_min = mdbx_datalen_min();
datalen_max = std::min(mdbx_datalen_max(), 256u * 1024 + 42);
batch_read = 4;
batch_write = 4;
@@ -122,9 +128,8 @@ int main(int argc, char *const argv[]) {
if (argc < 2)
failure("No parameters given\n");
if (argc == 2 &&
strncmp(argv[1], global::thunk_param_prefix,
strlen(global::thunk_param_prefix)) == 0)
if (argc == 2 && strncmp(argv[1], global::thunk_param_prefix,
strlen(global::thunk_param_prefix)) == 0)
return test_execute(
actor_config(argv[1] + strlen(global::thunk_param_prefix)))
? EXIT_SUCCESS
@@ -149,10 +154,53 @@ int main(int argc, char *const argv[]) {
config::mode_bits))
continue;
if (config::parse_option(argc, argv, narg, "table", params.table_flags,
config::table_bits))
config::table_bits)) {
if ((params.table_flags & MDBX_DUPFIXED) == 0)
params.table_flags &= ~MDBX_INTEGERDUP;
if ((params.table_flags & MDBX_DUPSORT) == 0)
params.table_flags &=
~(MDBX_DUPFIXED | MDBX_REVERSEDUP | MDBX_INTEGERDUP);
continue;
if (config::parse_option(argc, argv, narg, "size", params.size,
config::binary, 4096 * 4))
}
if (config::parse_option(argc, argv, narg, "pagesize", params.pagesize,
mdbx_limits_pgsize_min(),
mdbx_limits_pgsize_max())) {
const unsigned keylen_max = params.mdbx_keylen_max();
if (params.keylen_min > keylen_max)
params.keylen_min = keylen_max;
if (params.keylen_max > keylen_max)
params.keylen_max = keylen_max;
const unsigned datalen_max = params.mdbx_datalen_max();
if (params.datalen_min > datalen_max)
params.datalen_min = datalen_max;
if (params.datalen_max > datalen_max)
params.datalen_max = datalen_max;
continue;
}
if (config::parse_option(argc, argv, narg, "size-lower", params.size_lower,
mdbx_limits_dbsize_min(params.pagesize),
mdbx_limits_dbsize_max(params.pagesize)))
continue;
if (config::parse_option(argc, argv, narg, "size", params.size_now,
mdbx_limits_dbsize_min(params.pagesize),
mdbx_limits_dbsize_max(params.pagesize)))
continue;
if (config::parse_option(argc, argv, narg, "size-upper", params.size_upper,
mdbx_limits_dbsize_min(params.pagesize),
mdbx_limits_dbsize_max(params.pagesize)))
continue;
if (config::parse_option(
argc, argv, narg, "shrink-threshold", params.shrink_threshold, 0,
(int)std::min((intptr_t)INT_MAX,
mdbx_limits_dbsize_max(params.pagesize) -
mdbx_limits_dbsize_min(params.pagesize))))
continue;
if (config::parse_option(
argc, argv, narg, "growth-step", params.growth_step, 0,
(int)std::min((intptr_t)INT_MAX,
mdbx_limits_dbsize_max(params.pagesize) -
mdbx_limits_dbsize_min(params.pagesize))))
continue;
if (config::parse_option(argc, argv, narg, "keygen.width",
@@ -189,20 +237,39 @@ int main(int argc, char *const argv[]) {
config::duration, 1))
continue;
if (config::parse_option(argc, argv, narg, "keylen.min", params.keylen_min,
config::no_scale, 0, params.keylen_max))
config::no_scale, params.mdbx_keylen_min(),
params.mdbx_keylen_max())) {
if ((params.table_flags & MDBX_INTEGERKEY) ||
params.keylen_max < params.keylen_min)
params.keylen_max = params.keylen_min;
continue;
}
if (config::parse_option(argc, argv, narg, "keylen.max", params.keylen_max,
config::no_scale, params.keylen_min,
mdbx_get_maxkeysize(0)))
config::no_scale, params.mdbx_keylen_min(),
params.mdbx_keylen_max())) {
if ((params.table_flags & MDBX_INTEGERKEY) ||
params.keylen_min > params.keylen_max)
params.keylen_min = params.keylen_max;
continue;
}
if (config::parse_option(argc, argv, narg, "datalen.min",
params.datalen_min, config::no_scale, 0,
params.datalen_max))
params.datalen_min, config::no_scale,
params.mdbx_datalen_min(),
params.mdbx_datalen_max())) {
if ((params.table_flags & MDBX_DUPFIXED) ||
params.datalen_max < params.datalen_min)
params.datalen_max = params.datalen_min;
continue;
}
if (config::parse_option(argc, argv, narg, "datalen.max",
params.datalen_max, config::no_scale,
params.datalen_min, MDBX_MAXDATASIZE))
params.mdbx_datalen_min(),
params.mdbx_datalen_max())) {
if ((params.table_flags & MDBX_DUPFIXED) ||
params.datalen_min > params.datalen_max)
params.datalen_min = params.datalen_max;
continue;
}
if (config::parse_option(argc, argv, narg, "batch.read", params.batch_read,
config::no_scale, 1))
continue;

View File

@@ -182,6 +182,9 @@ void osal_killall_actors(void) {
}
int osal_actor_poll(mdbx_pid_t &pid, unsigned timeout) {
struct timespec ts;
ts.tv_nsec = 0;
ts.tv_sec = timeout;
retry:
int status, options = WNOHANG;
#ifdef WUNTRACED
@@ -209,9 +212,16 @@ retry:
}
if (pid == 0) {
if (timeout && sleep(timeout))
/* child still running */
if (ts.tv_sec == 0 && ts.tv_nsec == 0)
ts.tv_nsec = 1;
if (nanosleep(&ts, &ts) == 0) {
/* timeout and no signal fomr child */
pid = 0;
return 0;
}
if (errno == EINTR)
goto retry;
return 0;
}
switch (errno) {

View File

@@ -53,7 +53,7 @@ void osal_wait4barrier(void) {
}
}
static HANDLE make_inharitable(HANDLE hHandle) {
static HANDLE make_inheritable(HANDLE hHandle) {
assert(hHandle != NULL && hHandle != INVALID_HANDLE_VALUE);
if (!DuplicateHandle(GetCurrentProcess(), hHandle, GetCurrentProcess(),
&hHandle, 0, TRUE,
@@ -71,7 +71,7 @@ void osal_setup(const std::vector<actor_config> &actors) {
HANDLE hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (!hEvent)
failure_perror("CreateEvent()", GetLastError());
hEvent = make_inharitable(hEvent);
hEvent = make_inheritable(hEvent);
log_trace("osal_setup: event %" PRIuPTR " -> %p", i, hEvent);
events[i] = hEvent;
}
@@ -79,12 +79,12 @@ void osal_setup(const std::vector<actor_config> &actors) {
hBarrierSemaphore = CreateSemaphore(NULL, 0, (LONG)actors.size(), NULL);
if (!hBarrierSemaphore)
failure_perror("CreateSemaphore(BarrierSemaphore)", GetLastError());
hBarrierSemaphore = make_inharitable(hBarrierSemaphore);
hBarrierSemaphore = make_inheritable(hBarrierSemaphore);
hBarrierEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (!hBarrierEvent)
failure_perror("CreateEvent(BarrierEvent)", GetLastError());
hBarrierEvent = make_inharitable(hBarrierEvent);
hBarrierEvent = make_inheritable(hBarrierEvent);
}
void osal_broadcast(unsigned id) {
@@ -168,6 +168,90 @@ bool actor_config::osal_deserialize(const char *str, const char *end,
typedef std::pair<HANDLE, actor_status> child;
static std::unordered_map<mdbx_pid_t, child> childs;
static void ArgvQuote(std::string &CommandLine, const std::string &Argument,
bool Force = false)
/*++
https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/
Routine Description:
This routine appends the given argument to a command line such
that CommandLineToArgvW will return the argument string unchanged.
Arguments in a command line should be separated by spaces; this
function does not add these spaces.
Arguments:
Argument - Supplies the argument to encode.
CommandLine - Supplies the command line to which we append the encoded
argument string.
Force - Supplies an indication of whether we should quote
the argument even if it does not contain any characters that would
ordinarily require quoting.
Return Value:
None.
Environment:
Arbitrary.
--*/
{
//
// Unless we're told otherwise, don't quote unless we actually
// need to do so --- hopefully avoid problems if programs won't
// parse quotes properly
//
if (Force == false && Argument.empty() == false &&
Argument.find_first_of(" \t\n\v\"") == Argument.npos) {
CommandLine.append(Argument);
} else {
CommandLine.push_back('"');
for (auto It = Argument.begin();; ++It) {
unsigned NumberBackslashes = 0;
while (It != Argument.end() && *It == '\\') {
++It;
++NumberBackslashes;
}
if (It == Argument.end()) {
//
// Escape all backslashes, but let the terminating
// double quotation mark we add below be interpreted
// as a metacharacter.
//
CommandLine.append(NumberBackslashes * 2, '\\');
break;
} else if (*It == L'"') {
//
// Escape all backslashes and the following
// double quotation mark.
//
CommandLine.append(NumberBackslashes * 2 + 1, '\\');
CommandLine.push_back(*It);
} else {
//
// Backslashes aren't special here.
//
CommandLine.append(NumberBackslashes, '\\');
CommandLine.push_back(*It);
}
}
CommandLine.push_back('"');
}
}
int osal_actor_start(const actor_config &config, mdbx_pid_t &pid) {
if (childs.size() == MAXIMUM_WAIT_OBJECTS)
failure("Could't manage more that %u actors on Windows\n",
@@ -178,13 +262,23 @@ int osal_actor_start(const actor_config &config, mdbx_pid_t &pid) {
STARTUPINFOA StartupInfo;
GetStartupInfoA(&StartupInfo);
char exename[_MAX_PATH];
char exename[_MAX_PATH + 1];
DWORD exename_size = sizeof(exename);
if (!QueryFullProcessImageNameA(GetCurrentProcess(), 0, exename,
&exename_size))
failure_perror("QueryFullProcessImageName()", GetLastError());
std::string cmdline = "test_mdbx.child " + thunk_param(config);
if (exename[1] != ':') {
exename_size = GetModuleFileName(NULL, exename, sizeof(exename));
if (exename_size >= sizeof(exename))
return ERROR_BAD_LENGTH;
}
std::string cmdline = "$ ";
ArgvQuote(cmdline, thunk_param(config));
if (cmdline.size() >= 32767)
return ERROR_BAD_LENGTH;
PROCESS_INFORMATION ProcessInformation;
if (!CreateProcessA(exename, const_cast<char *>(cmdline.c_str()),
@@ -195,7 +289,7 @@ int osal_actor_start(const actor_config &config, mdbx_pid_t &pid) {
NULL, // Inherit the parent's environment.
NULL, // Inherit the parent's current directory.
&StartupInfo, &ProcessInformation))
return GetLastError();
failure_perror(exename, GetLastError());
CloseHandle(ProcessInformation.hThread);
pid = ProcessInformation.dwProcessId;

View File

@@ -68,31 +68,6 @@ const char *keygencase2str(const keygen_case keycase) {
//-----------------------------------------------------------------------------
static void mdbx_logger(int type, const char *function, int line,
const char *msg, va_list args) {
logging::loglevel level = logging::info;
if (type & MDBX_DBG_EXTRA)
level = logging::extra;
if (type & MDBX_DBG_TRACE)
level = logging::trace;
if (type & MDBX_DBG_PRINT)
level = logging::verbose;
if (!function)
function = "unknown";
if (type & MDBX_DBG_ASSERT) {
log_error("mdbx: assertion failure: %s, %d", function, line);
level = logging::failure;
}
if (logging::output(level, strncmp(function, "mdbx_", 5) == 0 ? "%s: "
: "mdbx: %s: ",
function))
logging::feed(msg, args);
if (type & MDBX_DBG_ASSERT)
abort();
}
int testcase::oom_callback(MDBX_env *env, int pid, mdbx_tid_t tid, uint64_t txn,
unsigned gap, int retry) {
@@ -117,16 +92,8 @@ void testcase::db_prepare() {
log_trace(">> db_prepare");
assert(!db_guard);
int mdbx_dbg_opts = MDBX_DBG_ASSERT | MDBX_DBG_JITTER | MDBX_DBG_DUMP;
if (config.params.loglevel <= logging::trace)
mdbx_dbg_opts |= MDBX_DBG_TRACE;
if (config.params.loglevel <= logging::verbose)
mdbx_dbg_opts |= MDBX_DBG_PRINT;
int rc = mdbx_setup_debug(mdbx_dbg_opts, mdbx_logger);
log_trace("set mdbx debug-opts: 0x%02x", rc);
MDBX_env *env = nullptr;
rc = mdbx_env_create(&env);
int rc = mdbx_env_create(&env);
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_env_create()", rc);
@@ -149,7 +116,10 @@ void testcase::db_prepare() {
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_env_set_oomfunc()", rc);
rc = mdbx_env_set_mapsize(env, (size_t)config.params.size);
rc = mdbx_env_set_geometry(
env, config.params.size_lower, config.params.size_now,
config.params.size_upper, config.params.growth_step,
config.params.shrink_threshold, config.params.pagesize);
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_env_set_mapsize()", rc);

View File

@@ -78,21 +78,25 @@
<LinkIncremental>true</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<LinkIncremental>true</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<LinkIncremental>false</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>

View File

@@ -93,7 +93,7 @@ bool hex2data(const char *hex_begin, const char *hex_end, void *ptr,
//-----------------------------------------------------------------------------
/* TODO: replace my 'libmera' fomr t1ha. */
/* TODO: replace my 'libmera' from t1ha. */
uint64_t entropy_ticks(void) {
#if defined(EMSCRIPTEN)
return (uint64_t)emscripten_get_now();