Compare commits

...

83 Commits

Author SHA1 Message Date
Leonid Yuriev
5c488d7033 mdbx: backport - fix pwrite() for WRITE_MAX.
Change-Id: If4924d20c1e267c2d3a190c860b89fc2fda0d517
2019-06-24 02:45:05 +03:00
Leonid Yuriev
5413407f23 mdbx-test: backport - fix dbsize-options handling.
Change-Id: Ia51f802ac1ad4e8b1b059a3f3b38214bda6b43fc
2019-06-24 02:45:05 +03:00
Leonid Yuriev
ebc8e9935e mdbx: bump version to v0.1.7
Change-Id: I0f72ed31fbd1ed74a875c2aa2023521855e72894
2019-06-22 23:31:10 +03:00
Leonid Yuriev
26838a2164 mdbx: rework RECLAIMING inside update_gc().
Change-Id: I9cf592476780bfdb346472baa12497d68a3d5aad
2019-06-22 23:31:05 +03:00
Howard Chu
828889de5c mdbx: import - tweak mdb_page_split (ITS#8969).
Bump up number of keys for which we use fine-grained splitpoint search

Change-Id: Icca2e1953cbcd6898b790f657636c2195b397790
2019-06-22 15:50:28 +03:00
Leonid Yuriev
6c160d02af mdbx: backport - fix TAGRET typo (minor).
Change-Id: Iffafbed7fdad3492aeb51f17caf8109a5b3e35c0
2019-06-22 01:48:12 +03:00
Leonid Yuriev
fead1c3853 mdbx: backport - fix handling MDBX_APPENDDUP mode.
Change-Id: I36de2a8dcab5126dab3857a7840ab3904a1d19c8
2019-06-22 01:48:12 +03:00
Leo Yuriev
06c35dd59c mdbx: backport - fix __ANDROID__ typo.
Thank to Howard Chu <hyc@openldap.org>.

Change-Id: Ibcbe2e4790a5df5758d9fd6c621793ea42a94682
2019-06-22 01:48:12 +03:00
Leonid Yuriev
46eb178f07 mdbx: backport - fix GC corruption due deep recursive rebalance from update_gc().
Change-Id: I810250deb25cd625e737000282b434e3158ef8cc
2019-06-22 01:32:47 +03:00
Leo Yuriev
b1ffe87556 mdbx: fix one more comment typo (minor). 2019-02-13 20:27:20 +03:00
Leo Yuriev
131485e516 mdbx: fix comment typo (minor). 2019-02-13 20:27:14 +03:00
Leo Yuriev
64f6648d0c mdbx: make API compatible to the master branch.
Change-Id: I95c5db639cebe4bba9c600f97c9966082bc9bc09
2019-02-03 13:17:43 +03:00
Leonid Yuriev
ba00b597a7 mdbx-windows: backport - fix returning negative value on failure.
Change-Id: Iaf5fb1f0cbcc3c14e2d3edf1f57538ecc0dfdf00
2019-02-03 13:17:43 +03:00
Leo Yuriev
4f79e3756c mdbx: 2019 HNY.
Change-Id: Iec6b7affedef0ea89fad917eb17af7e8201a7482
2019-02-03 13:17:43 +03:00
Leo Yuriev
f183cef7d7 mdbx-windows: backport - always susppend local threads while resize DB (workaround for Windows kernel bug).
We should not concern about performance on Windows platform,
it just unreasonable. Therefore just always suspend the local
threads to avoid this issue.

This resolves https://github.com/leo-yuriev/libmdbx/issues/48

Change-Id: I6e652692794b8c4c0d41625be62f2051b63c033a
2018-11-22 22:39:04 +03:00
Leonid Yuriev
850fe8408e mdbx: backport - relax DBI-sequences for concurrent open DBI-handles for present tables.
Change-Id: I7f07d2e716074bd9c2847aeb062e366f46cca214
2018-11-20 14:29:20 +03:00
Leonid Yuriev
0043f62a43 mdbx: backport - silently put mm_geo.now into [geo.lower...geo.upper] boundaries.
Copy-with-compaction by previous version of libmfbx could produce
DB-file less than meta.geo.lower bound, in case actual filling
is low or no data at all.

This is not a problem as there is no damage or loss of data.
Therefore it is better not to consider such situation as an
error, but silently correct it.

Change-Id: Ia662656cc3584c07efcfbdfc80f80e3c76e6dd59
2018-11-05 14:35:24 +03:00
Leonid Yuriev
de8d0479ab mdbx: backport - fix typo in mdbx_limits_dbsize_max().
Change-Id: Ie55e3ca108ac6aab9a41d65f316a3d5ff5ff6f1f
2018-11-05 02:16:29 +03:00
Leo Yuriev
feab109c61 mdbx-test: backport - fix osal_actor_poll() for 32-bit builds.
Change-Id: I36b2f955295d8ca5435a68737c0c2e7f069bfe34
2018-11-05 02:16:24 +03:00
Leonid Yuriev
6120c2be0a mdbx-test: backport - add checks for db-copy after the basic testcase.
Change-Id: I5e7d343266c66418a8798d272e697e1c3d5c775b
2018-11-05 02:16:19 +03:00
Leonid Yuriev
ee0c8bb249 mdbx: backport - add db-copy testcase.
Change-Id: Ib554880ebbabcb5dfc55bdb3c71767d0fa1630fd
2018-11-05 02:16:13 +03:00
Leonid Yuriev
3d59c9f9e7 mdbx: backport - take in account shrink/growing thresholds while copy-with-compactification.
Change-Id: Id93e62089819dfcc8cbc83620e0bdd806d8c1950
2018-11-05 02:16:07 +03:00
Leonid Yuriev
86e63f0b6b mdbx: backport - refine mdbx_env_copy() internals (required for next patch).
Change-Id: I9e8f0dc87398564524a5ec98eda2cb9bde100909
2018-11-05 02:16:07 +03:00
Leonid Yuriev
affd28654c mdbx: backport - fix mdbx_txn_abort().
This resolves https://github.com/leo-yuriev/libfpta/issues/20

Change-Id: I43c0c960d5c871d837b307cd370ee7327db01ff6
2018-10-22 01:45:45 +03:00
Leonid Yuriev
08130df595 mdbx-windows: backport - workaround for Windows10 bugs.
This resolves https://github.com/leo-yuriev/libmdbx/issues/47

Change-Id: I6e0d6dfbfec15b68200438b68a2996c357d46b77
2018-10-22 01:45:28 +03:00
Leonid Yuriev
5acf2b126f mdbx: backport - fix mdbx_dbi_sequence().
Change-Id: Ic620896ef42c1c2d85c07c146b72e773ab43a67d
2018-10-22 01:40:59 +03:00
Leonid Yuriev
cc84f85722 mdbx-ci: backport - disable CI for old MSVC compilers.
Change-Id: Ia1072745664d9a97d4114149a305a6399bde71aa
2018-10-22 01:37:29 +03:00
moneromooo-monero
a3aa2b5a57 mdbx-doc: import - mdb_cursor_del does not invalidate the cursor (ITS#8857). 2018-09-24 19:35:39 +03:00
Howard Chu
02276500c9 mdbx-doc: import - GET_MULTIPLE etc don't return the key (ITS#8908).
Unnecessary since these are DUPs, the key will always be the same
2018-09-24 19:35:36 +03:00
Leonid Yuriev
de44ecccd1 mdbx: backport - update MAX_PAGENO and MAX_MAPSIZE64. 2018-09-23 18:07:29 +03:00
Leonid Yuriev
5049c86517 mdbx: backport - avoid empty and unneeded large/overflow pages (squashed). 2018-09-23 18:07:29 +03:00
Leonid Yuriev
d2854e0760 mdbx: backport - refine mdbx_chk (squashed).
- refine 'mismatch idl length' error message.
 - add/fix printf-format checking.
 - refine dbi-structure.
2018-09-23 18:07:29 +03:00
Leonid Yuriev
5a29214ad9 mdbx-test: backport - update 'gc.sh' script (squashed). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
b51d92d449 mdbx-test: backport - update test (squashed).
- add support for 'default' options values.
 - add min/max cases for option values.
 - add support for db-geometry params.
 - fix int-types for 32-bit builds (minor).
 - fix key/value generation for long-length cases.
 - fix update_flags for non-MDBX_DUPSORT.
 - 'none' for config-verbs.
 - check commandline length under Windows.
 - workaround for QueryFullProcessImageNameA() bug.
 - add setloglevel().
 - workaroung for MSVC bug.
 - avoid extra 'jitter' testcase loops.
 - cleanup DUPSORT flags.
 - refine key/value min/max handling.
 - dump keygen params.
 - fix/refine keygen.
 - alter keygen defaults (rotate 3, offset 41).
 - default test-db size 4mb or 256mb.
 - fix/refine keygen for non-MDBX_DUPSORT.
 - seeding keygen with actor_id for better spreading.
2018-09-23 15:39:56 +03:00
Leonid Yuriev
6da477d37f mdbx-ci: backport - refines for Windows (squashed).
- push logs to appveyor separately.
 - rename 'test.exe' to 'mdbx_test.exe'.
 - add test.db to appveyor artefacts (windows).
2018-09-23 15:39:56 +03:00
Leonid Yuriev
6150a8c903 mdbx: backport - fix/refine mdbx_update_gc() (squashed). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
f3e9731da4 mdbx: backport - move macros/inlines to fix Windows builds.
Change-Id: I48aaf6b77466bb8b13294b84de73fb6063c88190
2018-09-23 15:39:56 +03:00
Leonid Yuriev
353b6b8af0 mdbx: backport - refine assections (minor).
Change-Id: Ic924988b8ce043d6106df381c996dd2c8ff9ca1f
2018-09-23 15:39:56 +03:00
Leonid Yuriev
3f10e58df2 mdbx: backport - re-define assert macro via mdbx_assert.
Change-Id: I317801ba4200bdf1aa5cacf75d21a8e633fbc48a
2018-09-23 15:39:56 +03:00
Leonid Yuriev
d232737087 mdbx: backport - add MDBX_FORCE_ASSERT.
Change-Id: I68a9f7b42663ea157c7c0a5a58797c94127b45ed
2018-09-23 15:39:56 +03:00
Leo Yuriev
e32ca55258 mdbx: backport - fix tracking around mdbx_cursor_del(). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
f57ffc987c mdbx: backport - drop inherited broken audit (will be fixed in the master branch).
Internal self-audit (inherited from LMDB) is invalid and useless
for sub-db and dupsort cases.
2018-09-23 15:39:56 +03:00
Leonid Yuriev
cdd510d20e mdbx: backport - prevent DB corruption due rebalance bugs.
Won't fix https://github.com/leo-yuriev/libmdbx/issues/38 in the 'stable/0.1' branch,
but add checks to prevent DB corruption.
2018-09-23 15:39:56 +03:00
Leo Yuriev
d757ba1266 mdbx: backport - fix MDBX_CORRUPTED due open/shrink collision. 2018-09-23 15:39:56 +03:00
Leonid Yuriev
337f7589f8 mdbx: backport - fix mdbx_pnl_search(). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
912728a322 mdbx: backport - fix mdbx_replace().
Change-Id: I2af00f101017795ca2b967479f86e5ea7e8ad37b
2018-09-23 15:39:56 +03:00
Leonid Yuriev
204b5a532d mdbx: backport - shorten maxkeysize (will be fixed in the master branch).
Change-Id: I660b1b3e454d9b51a24d3b4cc987c8e2980bd435
2018-09-23 15:39:56 +03:00
Leonid Yuriev
014be165c3 mdbx: backport - allow GC's PNL be partially unused. 2018-09-23 15:39:56 +03:00
Leonid Yuriev
a9244f807b mdbx: backport - setup mdbx_cmp_memn() as data-comparator for safety. 2018-09-23 15:39:56 +03:00
Leonid Yuriev
6d438605dd mdbx: backport - check comparator for MDBX_GET_BOTH and MDBX_GET_BOTH_RANGE. 2018-09-23 15:39:56 +03:00
Leo Yuriev
34300150a1 mdbx: backport - don't touch mm_psize and mm_flags while provoking bad readers (debug-only). 2018-09-23 15:39:56 +03:00
Leo Yuriev
1b2b98234f mdbx: backport - fix concurrent opening with custom pagesize (get pagesize from meta-page early). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
bd672a5583 mdbx: backport - add mdbx_limits_xyz() (squashed).
Change-Id: I56c79704c59386a0c4d84b001020484c23925e6c
2018-09-23 15:39:56 +03:00
Leonid Yuriev
888003c072 mdbx: backport - fix comments typos (squashed). 2018-09-23 14:40:31 +03:00
Leo Yuriev
bc6db4e4d7 mdbx: backport - allow mdbx_env_compact() to fix page leaks.
Don't treat fixing page leaks as an error while copy DB with compactification.

Change-Id: I2a575ff9e2b24610172aaca939b5f6957c26ec77
2018-09-23 14:40:31 +03:00
Leonid Yuriev
57655583e5 mdbx: backport - fix/rewrite mdbx_update_gc().
Change-Id: I580a1ff0cbeeb529e2bcbd50d97bfba7bcf5a546
2018-08-13 23:30:39 +03:00
Leonid Yuriev
b91e645919 mdbx-test: add 'gc.sh' script.
Change-Id: I633c93c0865b0d2609688713e986edf51ce6547d
2018-08-13 21:44:43 +03:00
Leonid Yuriev
652bb08f8c mdbx-test: backport - use strtoull() and retry with base=10.
Change-Id: Ica846ed0a13eb4468a45620518b9ccf85e77a764
2018-08-13 21:44:10 +03:00
Leonid Yuriev
59026d5f84 mdbx-test: backport - fix minor typos.
Change-Id: I4889a0e698bdfdda7eed257a5cd29e8b8089d102
2018-08-13 21:43:07 +03:00
Leonid Yuriev
e18551061e mdbx-test: backport - fix keylen/datalen min/max ranges checking.
Change-Id: Iee5d2f71ad22ec6e86167f5181deff54f0b5b518
2018-08-13 21:42:44 +03:00
Leonid Yuriev
e054ad2ebb mdbx-test: backport - add 'strikethrough' for bitmask-options.
Change-Id: I86dd2f8cdbd5a32a0471a5eee1e2b3a5857541ac
2018-08-13 21:42:22 +03:00
Leonid Yuriev
bff6aa460a mdbx: backport - fix MDBX_EKEYMISMATCH while update multi-value with MDBX_CURRENT.
Change-Id: I3095620a94f694fb2c29b9c4faab9ea02b9bd7b7
2018-08-13 21:41:17 +03:00
Leonid Yuriev
3979ba4784 mdbx: backport - fix assertions.
Change-Id: I95c43ef1ea2da55a124dc43f03890cf1d96f2e61
2018-08-13 21:40:48 +03:00
Leonid Yuriev
d2fcbf5f82 mdbx: backport - fix assert-condition inside mdbx_pnl_xappend().
Change-Id: Id5ac89c85b7e673c44d60a626c805fe666d221bc
2018-08-10 09:36:44 +03:00
Leo Yuriev
38067c4566 mdbx: backport - fix 'db_dummy' inside mdbx_dbi_open_ex().
Change-Id: I70a21c9b77a43c5af749da5723fa965487a056b0
2018-08-09 13:07:02 +03:00
Leo Yuriev
d4bfc17818 mdbx: backport - add fallback2shared for mdbx_lck_exclusive(). 2018-08-09 13:07:02 +03:00
Leo Yuriev
eb99480253 mdbx: backport - drop unused mdbx_lck_upgrade(). 2018-08-09 13:07:02 +03:00
Leo Yuriev
c9790b28d0 mdbx-cmake: fix so-version.
Change-Id: I427d2f27f9092d65a0ffd11353ca466070e98618
2018-08-09 13:07:02 +03:00
Leo Yuriev
9a1ef8acfb mdbx-cmake: remove warning-message.
Change-Id: Icf9e4f7a96916cf9ab04613344867217be04827a
2018-08-09 13:07:02 +03:00
Leo Yuriev
e442395cbd mdbx: bump version to v0.1.6
Change-Id: I95d45a815008e2cc9a8785a8c762310a1e907e21
2018-07-31 11:43:41 +03:00
Leo Yuriev
e57e521609 mdbx: backport - fix nasty suspend_and_append() bug.
Change-Id: I043adcff2e6c040426a51b5d4b15bac849e6dd9f
2018-07-31 11:43:35 +03:00
Leo Yuriev
d1809e6e2d mdbx: backport - minor fix to avoid Valgrind false-positive issue.
Change-Id: Ifa4dc51b500ff42a88182d750e22572aa5b2155b
2018-07-01 17:43:04 +03:00
Leo Yuriev
c579b974a2 mdbx: backport - avoid weak meta inside mdbx_init_metas().
Change-Id: Ib9c5ab04ad8cff3ad43d94a288cecec45d7ef37d
2018-06-30 02:43:10 +03:00
Leo Yuriev
de43ab0d21 mdbx-ci: migrate to Circle-CI 2.0
Change-Id: Id86af9e033d64a4dc2043db33cd8e7ae173feb22
2018-06-21 19:45:42 +03:00
Leo Yuriev
db50fb8726 mdbx: backport - fix Coverity warning (minor, paranoia).
Change-Id: I232377a03244dc33beb4f332c0024b454027f659
2018-06-21 18:26:31 +03:00
Leo Yuriev
408848a425 mdbx: Merge branch 'master' into stable/0.1
Change-Id: Ide2fbcd1b0b6bacbc4f07049633df81fede397eb
2018-06-21 17:57:06 +03:00
Leo Yuriev
e880e734ce mdbx: Merge branch 'master' into stable/0.1 2018-06-19 14:46:39 +03:00
Leo Yuriev
61b2a7fc54 mdbx: Merge branch 'master' into stable/0.1
Change-Id: I80b5afe1d227009a42096b5c6f2bfa9e5eb09036
2018-06-15 03:46:17 +03:00
Leo Yuriev
e1e17fd6a4 mdbx: Merge branch 'master' into stable/0.1 2018-06-14 13:54:01 +03:00
Leo Yuriev
17d3e7190c mdbx: Merge branch 'master' into stable/0.1
Change-Id: I7460e6a8b42c9bbeadfdf20d326c31c1d4a98969
2018-06-01 16:42:48 +03:00
Leo Yuriev
22d8b0b2e1 mdbx: Merge branch 'master' into stable/0.1
Change-Id: Idd1af2c07b48bfb1f335aa31a52bed4bd68514ac
2018-05-29 03:14:07 +03:00
Leo Yuriev
9384d0efa6 mdbx: Merge branch 'master' into stable/0.1
Change-Id: I3318b330e4aa5fd0db4642af6ef81e69814f0cc5
2018-05-22 12:03:57 +03:00
Leo Yuriev
ffafd5be10 mdbx: disable warning #5045 for MSVC (minor). 2018-05-21 16:36:41 +03:00
53 changed files with 1875 additions and 1164 deletions

20
.circleci/config.yml Normal file
View File

@@ -0,0 +1,20 @@
version: 2
jobs:
build:
docker:
- image: circleci/buildpack-deps:artful
environment:
- TESTDB: /tmp/test.db
- TESTLOG: /tmp/test.log
steps:
- checkout
- run: make all
- run: ulimit -c unlimited && make check
- run:
command: |
mkdir -p /tmp/artifacts
mv -t /tmp/artifacts $TESTLOG $TESTDB core.*
when: on_fail
- store_artifacts:
path: /tmp/artifacts
destination: test-artifacts

View File

@@ -2,19 +2,10 @@ cmake_minimum_required(VERSION 2.8.7)
set(TARGET mdbx)
project(${TARGET})
message(WARNING "
***************************************************************
MDBX is under active development, database format and API
aren't stable at least until 2018Q3. New version won't be
backwards compatible. Main focus of the rework is to provide
clear and robust API and new features.
***************************************************************
")
set(MDBX_VERSION_MAJOR 0)
set(MDBX_VERSION_MINOR 1)
set(MDBX_VERSION_RELEASE 3)
set(MDBX_VERSION_REVISION 1)
set(MDBX_VERSION_RELEASE 6)
set(MDBX_VERSION_REVISION 0)
set(MDBX_VERSION_STRING ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}.${MDBX_VERSION_RELEASE})
@@ -111,14 +102,14 @@ add_library(${TARGET}_SHARED SHARED
set_target_properties(${TARGET}_SHARED PROPERTIES
VERSION ${MDBX_VERSION_STRING}
SOVERSION ${MDBX_VERSION_MAJOR}
SOVERSION ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}
OUTPUT_NAME ${TARGET}
CLEAN_DIRECT_OUTPUT 1
)
set_target_properties(${TARGET}_STATIC PROPERTIES
VERSION ${MDBX_VERSION_STRING}
SOVERSION ${MDBX_VERSION_MAJOR}
SOVERSION ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}
OUTPUT_NAME ${TARGET}
CLEAN_DIRECT_OUTPUT 1
)

View File

@@ -1,4 +1,4 @@
Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>.
Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>.
Copyright 2011-2015 Howard Chu, Symas Corp.
Copyright 2015,2016 Peter-Service R&D LLC.
All rights reserved.

View File

@@ -82,13 +82,16 @@ clean:
rm -rf $(TOOLS) mdbx_test @* *.[ao] *.[ls]o *~ tmp.db/* *.gcov *.log *.err src/*.o test/*.o
check: all
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; ./mdbx_test --pathname=$(TESTDB) --dont-cleanup-after basic | tee -a $(TESTLOG) | tail -n 42) && ./mdbx_chk -vvn $(TESTDB)
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; ./mdbx_test --pathname=$(TESTDB) --dont-cleanup-after basic | tee -a $(TESTLOG) | tail -n 42) \
&& ./mdbx_chk -vvn $(TESTDB) && ./mdbx_chk -vvn $(TESTDB)-copy
check-singleprocess: all
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; ./mdbx_test --pathname=$(TESTDB) --dont-cleanup-after --hill | tee -a $(TESTLOG) | tail -n 42) && ./mdbx_chk -vvn $(TESTDB)
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; ./mdbx_test --pathname=$(TESTDB) --dont-cleanup-after --hill | tee -a $(TESTLOG) | tail -n 42) \
&& ./mdbx_chk -vvn $(TESTDB) && ./mdbx_chk -vvn $(TESTDB)-copy
check-fault: all
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; ./mdbx_test --pathname=$(TESTDB) --inject-writefault=42 --dump-config --dont-cleanup-after basic | tee -a $(TESTLOG) | tail -n 42) && ./mdbx_chk -vvn $(TESTDB)
rm -f $(TESTDB) $(TESTLOG) && (set -o pipefail; ./mdbx_test --pathname=$(TESTDB) --inject-writefault=42 --dump-config --dont-cleanup-after basic | tee -a $(TESTLOG) | tail -n 42) \
&& ./mdbx_chk -vvn $(TESTDB) && ./mdbx_chk -vvn $(TESTDB)-copy
define core-rule
$(patsubst %.c,%.o,$(1)): $(1) $(CORE_INC) mdbx.h Makefile

View File

@@ -12,24 +12,21 @@ and [by Yandex](https://translate.yandex.ru/translate?url=https%3A%2F%2Fgithub.c
### Project Status
**Сейчас MDBX _активно перерабатывается_** предстоит
большое изменение как API, так и формата базы данных. К сожалению,
обновление приведет к потере совместимости с предыдущими версиями.
**Сейчас MDBX _активно перерабатывается_** и к середине 2018
ожидается большое изменение как API, так и формата базы данных.
К сожалению, обновление приведет к потере совместимости с
предыдущими версиями.
Цель этой революции - обеспечение более четкого надежного API и
добавление новых функции, а также наделение базы данных новыми
свойствами.
Цель этой революции - обеспечение более четкого надежного
API и добавление новых функции, а также наделение базы данных
новыми свойствами.
В настоящее время MDBX предназначена для Linux, а также
поддерживает Windows (начиная с Windows Server 2008) в качестве
дополнительной платформы. Поддержка других ОС может быть
обеспечена на коммерческой основе. Однако такие
усовершенствования (т. е. pull-requests) могут быть приняты в
мейнстрим только в том случае, если будет доступен
соответствующий публичный и бесплатный сервис непрерывной
интеграции (aka Continuous Integration).
В настоящее время MDBX предназначена для Linux, а также поддерживает
Windows (начиная с Windows Server 2008) в качестве дополнительной
платформы. Поддержка других ОС может быть обеспечена на коммерческой
основе. Однако такие усовершенствования (т. е. pull-requests) могут быть
приняты в мейнстрим только в том случае, если будет доступен
соответствующий публичный и бесплатный сервис непрерывной интеграции
(aka Continuous Integration).
## Содержание

View File

@@ -1,13 +1,13 @@
version: 0.1.5.{build}
version: 0.1.6.{build}
environment:
matrix:
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017
TOOLSET: v141
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
TOOLSET: v140
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2013
TOOLSET: v120
# - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
# TOOLSET: v140
# - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2013
# TOOLSET: v120
branches:
except:
@@ -32,21 +32,23 @@ build_script:
test_script:
- ps: |
if (($env:PLATFORM -eq "x86") -and (Test-Path "C:\projects\libmdbx\Win32\$env:CONFIGURATION\test.exe" -PathType Leaf)) {
$test = "C:\projects\libmdbx\Win32\$env:CONFIGURATION\test.exe"
if (($env:PLATFORM -eq "x86") -and (Test-Path "C:\projects\libmdbx\Win32\$env:CONFIGURATION\mdbx_test.exe" -PathType Leaf)) {
$mdbx_test = "C:\projects\libmdbx\Win32\$env:CONFIGURATION\mdbx_test.exe"
$mdbx_chk = "C:\projects\libmdbx\Win32\$env:CONFIGURATION\mdbx_chk.exe"
} elseif (($env:PLATFORM -ne "ARM") -and ($env:PLATFORM -ne "ARM64")) {
$test = "C:\projects\libmdbx\$env:PLATFORM\$env:CONFIGURATION\test.exe"
$mdbx_test = "C:\projects\libmdbx\$env:PLATFORM\$env:CONFIGURATION\mdbx_test.exe"
$mdbx_chk = "C:\projects\libmdbx\$env:PLATFORM\$env:CONFIGURATION\mdbx_chk.exe"
} else {
$test = ""
$mdbx_test = ""
$mdbx_chk = ""
}
if ($test -ne "") {
& "$test" --pathname=tmp.db --dont-cleanup-after basic | Tee-Object -file test.log | Select-Object -last 42
& "$mdbx_chk" -nvv tmp.db | Tee-Object -file chk.log | Select-Object -last 42
if ($mdbx_test -ne "") {
& "$mdbx_test" --pathname=test.db --dont-cleanup-after basic | Tee-Object -file test.log | Select-Object -last 42
& "$mdbx_chk" -nvv test.db | Tee-Object -file chk.log | Select-Object -last 42
}
on_failure:
- ps: Push-AppveyorArtifact test.log chk.log
- ps: Push-AppveyorArtifact test.log
- ps: Push-AppveyorArtifact test.db
- ps: Push-AppveyorArtifact chk.log

View File

@@ -1,14 +0,0 @@
machine:
timezone:
Europe/Moscow
database:
override:
compile:
override:
- make all
test:
override:
- make check || mv test.log ${CIRCLE_ARTIFACTS}/

View File

@@ -3,6 +3,7 @@ README-RU.md
pcrf_test/CMakeLists.txt
src/tools/CMakeLists.txt
test/CMakeLists.txt
test/copy.cc
tutorial/CMakeLists.txt
tutorial/sample-mdbx.c
AUTHORS

39
mdbx.h
View File

@@ -1,6 +1,6 @@
/* LICENSE AND COPYRUSTING *****************************************************
*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -60,8 +60,8 @@
/* IMPENDING CHANGES WARNING ***************************************************
*
* MDBX is under active development, database format and API aren't stable
* at least until 2018Q2. New version won't be backwards compatible. Main focus
* MDBX is under active non-public development, database format and API
* will be refined. New version won't be backwards compatible. Main focus
* of the rework is to provide clear and robust API and new features.
*
******************************************************************************/
@@ -100,6 +100,7 @@ typedef DWORD mdbx_tid_t;
#define MDBX_EIO ERROR_WRITE_FAULT
#define MDBX_EPERM ERROR_INVALID_FUNCTION
#define MDBX_EINTR ERROR_CANCELLED
#define MDBX_ENOFILE ERROR_FILE_NOT_FOUND
#else
@@ -120,6 +121,8 @@ typedef pthread_t mdbx_tid_t;
#define MDBX_EIO EIO
#define MDBX_EPERM EPERM
#define MDBX_EINTR EINTR
#define MDBX_ENOFILE ENOENT
#endif
#ifdef _MSC_VER
@@ -344,17 +347,17 @@ typedef enum MDBX_cursor_op {
MDBX_GET_BOTH, /* MDBX_DUPSORT-only: Position at key/data pair. */
MDBX_GET_BOTH_RANGE, /* MDBX_DUPSORT-only: position at key, nearest data. */
MDBX_GET_CURRENT, /* Return key/data at current cursor position */
MDBX_GET_MULTIPLE, /* MDBX_DUPFIXED-only: Return key and up to a page of
* duplicate data items from current cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE.*/
MDBX_GET_MULTIPLE, /* MDBX_DUPFIXED-only: Return up to a page of duplicate
* data items from current cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE. */
MDBX_LAST, /* Position at last key/data item */
MDBX_LAST_DUP, /* MDBX_DUPSORT-only: Position at last data item
* of current key. */
MDBX_NEXT, /* Position at next data item */
MDBX_NEXT_DUP, /* MDBX_DUPSORT-only: Position at next data item
* of current key. */
MDBX_NEXT_MULTIPLE, /* MDBX_DUPFIXED-only: Return key and up to a page of
* duplicate data items from next cursor position.
MDBX_NEXT_MULTIPLE, /* MDBX_DUPFIXED-only: Return up to a page of duplicate
* data items from next cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE. */
MDBX_NEXT_NODUP, /* Position at first data item of next key */
MDBX_PREV, /* Position at previous data item */
@@ -366,7 +369,7 @@ typedef enum MDBX_cursor_op {
MDBX_SET_RANGE, /* Position at first key greater than or equal to
* specified key. */
MDBX_PREV_MULTIPLE /* MDBX_DUPFIXED-only: Position at previous page and
* return key and up to a page of duplicate data items. */
* return up to a page of duplicate data items. */
} MDBX_cursor_op;
/* Return Codes
@@ -468,7 +471,7 @@ typedef struct MDBX_envinfo {
uint64_t lower; /* lower limit for datafile size */
uint64_t upper; /* upper limit for datafile size */
uint64_t current; /* current datafile size */
uint64_t shrink; /* shrink theshold for datafile */
uint64_t shrink; /* shrink threshold for datafile */
uint64_t grow; /* growth step for datafile */
} mi_geo;
uint64_t mi_mapsize; /* Size of the data memory map */
@@ -588,7 +591,7 @@ LIBMDBX_API int mdbx_env_create(MDBX_env **penv);
* - MDBX_NOTLS
* Don't use Thread-Local Storage. Tie reader locktable slots to
* MDBX_txn objects instead of to threads. I.e. mdbx_txn_reset() keeps
* the slot reseved for the MDBX_txn object. A thread may use parallel
* the slot reserved for the MDBX_txn object. A thread may use parallel
* read-only transactions. A read-only transaction may span threads if
* the user synchronizes its use. Applications that multiplex many
* user threads over individual OS threads need this option. Such an
@@ -900,7 +903,7 @@ LIBMDBX_API int mdbx_env_set_maxdbs(MDBX_env *env, MDBX_dbi dbs);
*
* Returns The maximum size of a key we can write. */
LIBMDBX_API int mdbx_env_get_maxkeysize(MDBX_env *env);
LIBMDBX_API int mdbx_get_maxkeysize(size_t pagesize);
LIBMDBX_API int mdbx_get_maxkeysize(intptr_t pagesize);
/* Set application information associated with the MDBX_env.
*
@@ -966,7 +969,7 @@ LIBMDBX_API int mdbx_env_set_assert(MDBX_env *env, MDBX_assert_func *func);
* Returns A non-zero error value on failure and 0 on success, some
* possible errors are:
* - MDBX_PANIC - a fatal error occurred earlier and the environment
* must be shut down.
* must be shut down.
* - MDBX_MAP_RESIZED - another process wrote data beyond this MDBX_env's
* mapsize and this environment's map must be resized
* as well. See mdbx_env_set_mapsize().
@@ -1461,6 +1464,9 @@ LIBMDBX_API int mdbx_cursor_put(MDBX_cursor *cursor, MDBX_val *key,
/* Delete current key/data pair
*
* This function deletes the key/data pair to which the cursor refers.
* This does not invalidate the cursor, so operations such as MDBX_NEXT
* can still be used on it. Both MDBX_NEXT and MDBX_GET_CURRENT will return
* the same record after this operation.
*
* [in] cursor A cursor handle returned by mdbx_cursor_open()
* [in] flags Options for this operation. This parameter must be set to 0
@@ -1673,6 +1679,13 @@ LIBMDBX_API int mdbx_is_dirty(const MDBX_txn *txn, const void *ptr);
LIBMDBX_API int mdbx_dbi_sequence(MDBX_txn *txn, MDBX_dbi dbi, uint64_t *result,
uint64_t increment);
LIBMDBX_API int mdbx_limits_pgsize_min(void);
LIBMDBX_API int mdbx_limits_pgsize_max(void);
LIBMDBX_API intptr_t mdbx_limits_dbsize_min(intptr_t pagesize);
LIBMDBX_API intptr_t mdbx_limits_dbsize_max(intptr_t pagesize);
LIBMDBX_API intptr_t mdbx_limits_keysize_max(intptr_t pagesize);
LIBMDBX_API intptr_t mdbx_limits_txnsize_max(intptr_t pagesize);
/*----------------------------------------------------------------------------*/
/* attribute support functions for Nexenta */
typedef uint_fast64_t mdbx_attr_t;

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -160,7 +160,7 @@
* size up to 2^44 bytes, in case of 4K pages. */
typedef uint32_t pgno_t;
#define PRIaPGNO PRIu32
#define MAX_PAGENO ((pgno_t)UINT64_C(0xffffFFFFffff))
#define MAX_PAGENO UINT32_C(0x7FFFffff)
#define MIN_PAGENO NUM_METAS
/* A transaction ID. */
@@ -389,9 +389,7 @@ typedef struct MDBX_page {
#else
#define MAX_MAPSIZE32 UINT32_C(0x7ff80000)
#endif
#define MAX_MAPSIZE64 \
((sizeof(pgno_t) > 4) ? UINT64_C(0x7fffFFFFfff80000) \
: MAX_PAGENO * (uint64_t)MAX_PAGESIZE)
#define MAX_MAPSIZE64 (MAX_PAGENO * (uint64_t)MAX_PAGESIZE)
#define MAX_MAPSIZE ((sizeof(size_t) < 8) ? MAX_MAPSIZE32 : MAX_MAPSIZE64)
@@ -408,7 +406,7 @@ typedef struct MDBX_lockinfo {
volatile uint32_t mti_envmode;
#ifdef MDBX_OSAL_LOCK
/* Mutex protecting write access to this table. */
/* Mutex protecting write-txn. */
union {
MDBX_OSAL_LOCK mti_wmutex;
uint8_t pad_mti_wmutex[MDBX_OSAL_LOCK_SIZE % sizeof(size_t)];
@@ -751,8 +749,8 @@ struct MDBX_env {
MDBX_PNL me_free_pgs;
/* ID2L of pages written during a write txn. Length MDBX_PNL_UM_SIZE. */
MDBX_ID2L me_dirtylist;
/* Max number of freelist items that can fit in a single overflow page */
unsigned me_maxfree_1pg;
/* Number of freelist items that can fit in a single overflow page */
unsigned me_maxgc_ov1page;
/* Max size of a node on a page */
unsigned me_nodemax;
unsigned me_maxkey_limit; /* max size of a key */
@@ -817,15 +815,12 @@ void mdbx_panic(const char *fmt, ...)
#define mdbx_assert_enabled() unlikely(mdbx_runtime_flags &MDBX_DBG_ASSERT)
#define mdbx_audit_enabled() unlikely(mdbx_runtime_flags &MDBX_DBG_AUDIT)
#define mdbx_debug_enabled(type) \
unlikely(mdbx_runtime_flags &(type & (MDBX_DBG_TRACE | MDBX_DBG_EXTRA)))
#else
#define mdbx_debug_enabled(type) (0)
#define mdbx_audit_enabled() (0)
#ifndef NDEBUG
#if !defined(NDEBUG) || defined(MDBX_FORCE_ASSERT)
#define mdbx_assert_enabled() (1)
#else
#define mdbx_assert_enabled() (0)
@@ -931,17 +926,8 @@ void mdbx_panic(const char *fmt, ...)
/* assert(3) variant in transaction context */
#define mdbx_tassert(txn, expr) mdbx_assert((txn)->mt_env, expr)
static __inline void mdbx_jitter4testing(bool tiny) {
#ifndef NDEBUG
if (MDBX_DBG_JITTER & mdbx_runtime_flags)
mdbx_osal_jitter(tiny);
#else
(void)tiny;
#endif
}
/*----------------------------------------------------------------------------*/
/* Internal prototypes and inlines */
/* Internal prototypes */
int mdbx_reader_check0(MDBX_env *env, int rlocked, int *dead);
int mdbx_rthc_alloc(mdbx_thread_key_t *key, MDBX_reader *begin,
@@ -952,24 +938,6 @@ void mdbx_rthc_global_init(void);
void mdbx_rthc_global_dtor(void);
void mdbx_rthc_thread_dtor(void *ptr);
static __inline bool mdbx_is_power2(size_t x) { return (x & (x - 1)) == 0; }
static __inline size_t mdbx_roundup2(size_t value, size_t granularity) {
assert(mdbx_is_power2(granularity));
return (value + granularity - 1) & ~(granularity - 1);
}
static __inline unsigned mdbx_log2(size_t value) {
assert(mdbx_is_power2(value));
unsigned log = 0;
while (value > 1) {
log += 1;
value >>= 1;
}
return log;
}
#define MDBX_IS_ERROR(rc) \
((rc) != MDBX_RESULT_TRUE && (rc) != MDBX_RESULT_FALSE)
@@ -1043,6 +1011,8 @@ static __inline unsigned mdbx_log2(size_t value) {
/* Test if a page is a sub page */
#define IS_SUBP(p) F_ISSET((p)->mp_flags, P_SUBP)
#define PAGETYPE(p) ((p)->mp_flags & (P_BRANCH | P_LEAF | P_LEAF2 | P_OVERFLOW))
/* The number of overflow pages needed to store the given size. */
#define OVPAGES(env, size) (bytes2pgno(env, PAGEHDRSZ - 1 + (size)) + 1)
@@ -1113,71 +1083,12 @@ typedef struct MDBX_node {
* This is node header plus key plus data size. */
#define LEAFSIZE(k, d) (NODESIZE + (k)->iov_len + (d)->iov_len)
/* Address of node i in page p */
static __inline MDBX_node *NODEPTR(MDBX_page *p, unsigned i) {
assert(NUMKEYS(p) > (unsigned)(i));
return (MDBX_node *)((char *)(p) + (p)->mp_ptrs[i] + PAGEHDRSZ);
}
/* Address of the key for the node */
#define NODEKEY(node) (void *)((node)->mn_data)
/* Address of the data for a node */
#define NODEDATA(node) (void *)((char *)(node)->mn_data + (node)->mn_ksize)
/* Get the page number pointed to by a branch node */
static __inline pgno_t NODEPGNO(const MDBX_node *node) {
pgno_t pgno;
if (UNALIGNED_OK) {
pgno = node->mn_ksize_and_pgno;
if (sizeof(pgno_t) > 4)
pgno &= MAX_PAGENO;
} else {
pgno = node->mn_lo | ((pgno_t)node->mn_hi << 16);
if (sizeof(pgno_t) > 4)
pgno |= ((uint64_t)node->mn_flags) << 32;
}
return pgno;
}
/* Set the page number in a branch node */
static __inline void SETPGNO(MDBX_node *node, pgno_t pgno) {
assert(pgno <= MAX_PAGENO);
if (UNALIGNED_OK) {
if (sizeof(pgno_t) > 4)
pgno |= ((uint64_t)node->mn_ksize) << 48;
node->mn_ksize_and_pgno = pgno;
} else {
node->mn_lo = (uint16_t)pgno;
node->mn_hi = (uint16_t)(pgno >> 16);
if (sizeof(pgno_t) > 4)
node->mn_flags = (uint16_t)((uint64_t)pgno >> 32);
}
}
/* Get the size of the data in a leaf node */
static __inline size_t NODEDSZ(const MDBX_node *node) {
size_t size;
if (UNALIGNED_OK) {
size = node->mn_dsize;
} else {
size = node->mn_lo | ((size_t)node->mn_hi << 16);
}
return size;
}
/* Set the size of the data for a leaf node */
static __inline void SETDSZ(MDBX_node *node, size_t size) {
assert(size < INT_MAX);
if (UNALIGNED_OK) {
node->mn_dsize = (uint32_t)size;
} else {
node->mn_lo = (uint16_t)size;
node->mn_hi = (uint16_t)(size >> 16);
}
}
/* The size of a key in a node */
#define NODEKSZ(node) ((node)->mn_ksize)
@@ -1230,19 +1141,8 @@ static __inline void SETDSZ(MDBX_node *node, size_t size) {
#define mdbx_cmp2int(a, b) (((a) > (b)) - ((b) > (a)))
#endif
static __inline size_t pgno2bytes(const MDBX_env *env, pgno_t pgno) {
mdbx_assert(env, (1u << env->me_psize2log) == env->me_psize);
return ((size_t)pgno) << env->me_psize2log;
}
static __inline MDBX_page *pgno2page(const MDBX_env *env, pgno_t pgno) {
return (MDBX_page *)(env->me_map + pgno2bytes(env, pgno));
}
static __inline pgno_t bytes2pgno(const MDBX_env *env, size_t bytes) {
mdbx_assert(env, (env->me_psize >> env->me_psize2log) == 1);
return (pgno_t)(bytes >> env->me_psize2log);
}
/* Do not spill pages to disk if txn is getting full, may fail instead */
#define MDBX_NOSPILL 0x8000
static __inline pgno_t pgno_add(pgno_t base, pgno_t augend) {
assert(base <= MAX_PAGENO);
@@ -1254,10 +1154,11 @@ static __inline pgno_t pgno_sub(pgno_t base, pgno_t subtrahend) {
return (subtrahend < base - MIN_PAGENO) ? base - subtrahend : MIN_PAGENO;
}
static __inline size_t pgno_align2os_bytes(const MDBX_env *env, pgno_t pgno) {
return mdbx_roundup2(pgno2bytes(env, pgno), env->me_os_psize);
}
static __inline pgno_t pgno_align2os_pgno(const MDBX_env *env, pgno_t pgno) {
return bytes2pgno(env, pgno_align2os_bytes(env, pgno));
static __inline void mdbx_jitter4testing(bool tiny) {
#ifndef NDEBUG
if (MDBX_DBG_JITTER & mdbx_runtime_flags)
mdbx_osal_jitter(tiny);
#else
(void)tiny;
#endif
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -20,8 +20,8 @@
#ifndef MDBX_USE_ROBUST
/* Howard Chu: Android currently lacks Robust Mutex support */
#if defined(EOWNERDEAD) && \
!defined(ANDROID) /* LY: glibc before 2.10 has a troubles with Robust \
Mutex too. */ \
!defined(__ANDROID__) /* LY: glibc before 2.10 has a troubles \
with Robust Mutex too. */ \
&& __GLIBC_PREREQ(2, 10)
#define MDBX_USE_ROBUST 1
#else
@@ -68,11 +68,19 @@ static int mdbx_lck_op(mdbx_filehandle_t fd, int op, int lck, off_t offset,
}
}
static __inline int mdbx_lck_exclusive(int lfd) {
static __inline int mdbx_lck_exclusive(int lfd, bool fallback2shared) {
assert(lfd != INVALID_HANDLE_VALUE);
if (flock(lfd, LOCK_EX | LOCK_NB))
return errno;
return mdbx_lck_op(lfd, F_SETLK, F_WRLCK, 0, 1);
int rc = mdbx_lck_op(lfd, F_SETLK, F_WRLCK, 0, 1);
if (rc != 0 && fallback2shared) {
while (flock(lfd, LOCK_SH)) {
int rc = errno;
if (rc != EINTR)
return rc;
}
}
return rc;
}
static __inline int mdbx_lck_shared(int lfd) {
@@ -89,8 +97,6 @@ int mdbx_lck_downgrade(MDBX_env *env, bool complete) {
return complete ? mdbx_lck_shared(env->me_lfd) : MDBX_SUCCESS;
}
int mdbx_lck_upgrade(MDBX_env *env) { return mdbx_lck_exclusive(env->me_lfd); }
int mdbx_rpid_set(MDBX_env *env) {
return mdbx_lck_op(env->me_lfd, F_SETLK, F_WRLCK, env->me_pid, 1);
}
@@ -159,7 +165,7 @@ bailout:
void mdbx_lck_destroy(MDBX_env *env) {
if (env->me_lfd != INVALID_HANDLE_VALUE) {
/* try get exclusive access */
if (env->me_lck && mdbx_lck_exclusive(env->me_lfd) == 0) {
if (env->me_lck && mdbx_lck_exclusive(env->me_lfd, false) == 0) {
mdbx_info("%s: got exclusive, drown mutexes", mdbx_func_);
int rc = pthread_mutex_destroy(&env->me_lck->mti_rmutex);
if (rc == 0)
@@ -227,7 +233,7 @@ static int internal_seize_lck(int lfd) {
assert(lfd != INVALID_HANDLE_VALUE);
/* try exclusive access */
int rc = mdbx_lck_exclusive(lfd);
int rc = mdbx_lck_exclusive(lfd, false);
if (rc == 0)
/* got exclusive */
return MDBX_RESULT_TRUE;
@@ -236,7 +242,7 @@ static int internal_seize_lck(int lfd) {
rc = mdbx_lck_shared(lfd);
if (rc == 0) {
/* got shared, try exclusive again */
rc = mdbx_lck_exclusive(lfd);
rc = mdbx_lck_exclusive(lfd, true);
if (rc == 0)
/* now got exclusive */
return MDBX_RESULT_TRUE;

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -190,16 +190,25 @@ static int suspend_and_append(mdbx_handle_array_t **array,
(limit * 2 - ARRAY_LENGTH((*array)->handles)));
if (!ptr)
return MDBX_ENOMEM;
(*array) = (mdbx_handle_array_t *)ptr;
if (limit == ARRAY_LENGTH((*array)->handles))
memcpy(ptr, *array, sizeof(mdbx_handle_array_t));
*array = (mdbx_handle_array_t *)ptr;
(*array)->limit = limit * 2;
}
HANDLE hThread = OpenThread(THREAD_SUSPEND_RESUME, FALSE, ThreadId);
HANDLE hThread = OpenThread(THREAD_SUSPEND_RESUME | THREAD_QUERY_INFORMATION,
FALSE, ThreadId);
if (hThread == NULL)
return GetLastError();
if (SuspendThread(hThread) == -1) {
int err = GetLastError();
DWORD ExitCode;
if (err == /* workaround for Win10 UCRT bug */ ERROR_ACCESS_DENIED ||
!GetExitCodeThread(hThread, &ExitCode) || ExitCode != STILL_ACTIVE)
err = MDBX_SUCCESS;
CloseHandle(hThread);
return GetLastError();
return err;
}
(*array)->handles[(*array)->count++] = hThread;
@@ -283,9 +292,15 @@ int mdbx_suspend_threads_before_remap(MDBX_env *env,
int mdbx_resume_threads_after_remap(mdbx_handle_array_t *array) {
int rc = MDBX_SUCCESS;
for (unsigned i = 0; i < array->count; ++i) {
if (ResumeThread(array->handles[i]) == -1)
rc = GetLastError();
CloseHandle(array->handles[i]);
const HANDLE hThread = array->handles[i];
if (ResumeThread(hThread) == -1) {
const int err = GetLastError();
DWORD ExitCode;
if (err != /* workaround for Win10 UCRT bug */ ERROR_ACCESS_DENIED &&
GetExitCodeThread(hThread, &ExitCode) && ExitCode == STILL_ACTIVE)
rc = err;
}
CloseHandle(hThread);
}
return rc;
}
@@ -439,47 +454,6 @@ int mdbx_lck_downgrade(MDBX_env *env, bool complete) {
return MDBX_SUCCESS /* 7) now at S-? (used), done */;
}
int mdbx_lck_upgrade(MDBX_env *env) {
/* Transite from locked state (S-E) to exclusive-write (E-E) */
assert(env->me_fd != INVALID_HANDLE_VALUE);
assert(env->me_lfd != INVALID_HANDLE_VALUE);
/* 1) must be at S-E (locked), transite to ?_E (middle) */
if (!funlock(env->me_lfd, LCK_LOWER))
mdbx_panic("%s(%s) failed: errcode %u", mdbx_func_,
"S-E(locked) >> ?-E(middle)", GetLastError());
/* 3) now on ?-E (middle), try E-E (exclusive-write) */
mdbx_jitter4testing(false);
if (flock(env->me_lfd, LCK_EXCLUSIVE | LCK_DONTWAIT, LCK_LOWER))
return MDBX_RESULT_TRUE; /* 4) got E-E (exclusive-write), done */
/* 5) still on ?-E (middle) */
int rc = GetLastError();
mdbx_jitter4testing(false);
if (rc != ERROR_SHARING_VIOLATION && rc != ERROR_LOCK_VIOLATION) {
/* 6) something went wrong, report but continue */
mdbx_error("%s(%s) failed: errcode %u", mdbx_func_,
"?-E(middle) >> E-E(exclusive-write)", rc);
}
/* 7) still on ?-E (middle), try restore S-E (locked) */
mdbx_jitter4testing(false);
rc = flock(env->me_lfd, LCK_SHARED | LCK_DONTWAIT, LCK_LOWER)
? MDBX_RESULT_FALSE
: GetLastError();
mdbx_jitter4testing(false);
if (rc != MDBX_RESULT_FALSE) {
mdbx_fatal("%s(%s) failed: errcode %u", mdbx_func_,
"?-E(middle) >> S-E(locked)", rc);
return rc;
}
/* 8) now on S-E (locked) */
return MDBX_RESULT_FALSE;
}
void mdbx_lck_destroy(MDBX_env *env) {
int rc;

1645
src/mdbx.c

File diff suppressed because it is too large Load Diff

View File

@@ -1,7 +1,7 @@
/* https://en.wikipedia.org/wiki/Operating_system_abstraction_layer */
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -399,6 +399,13 @@ int mdbx_fastmutex_release(mdbx_fastmutex_t *fastmutex) {
/*----------------------------------------------------------------------------*/
int mdbx_removefile(const char *pathname) {
#if defined(_WIN32) || defined(_WIN64)
return DeleteFileA(pathname) ? MDBX_SUCCESS : GetLastError();
#else
return unlink(pathname) ? errno : MDBX_SUCCESS;
#endif
}
int mdbx_openfile(const char *pathname, int flags, mode_t mode,
mdbx_filehandle_t *fd) {
*fd = INVALID_HANDLE_VALUE;
@@ -526,17 +533,25 @@ int mdbx_pwrite(mdbx_filehandle_t fd, const void *buf, size_t bytes,
return (bytes == written) ? MDBX_SUCCESS : MDBX_EIO /* ERROR_WRITE_FAULT */;
return GetLastError();
#else
int rc;
intptr_t written;
do {
while (true) {
STATIC_ASSERT_MSG(sizeof(off_t) >= sizeof(size_t),
"libmdbx requires 64-bit file I/O on 64-bit systems");
written = pwrite(fd, buf, bytes, offset);
const intptr_t written =
pwrite(fd, buf, (bytes <= MAX_WRITE) ? bytes : MAX_WRITE, offset);
if (likely(bytes == (size_t)written))
return MDBX_SUCCESS;
rc = errno;
} while (rc == EINTR);
return (written < 0) ? rc : MDBX_EIO /* Use which error code (ENOSPC)? */;
if (written < 0) {
const int rc = errno;
if (rc != EINTR)
return rc;
} else if (written > 0) {
bytes -= written;
offset += written;
buf = (char *)buf + written;
} else {
return -1;
}
}
#endif
}
@@ -698,6 +713,19 @@ int mdbx_ftruncate(mdbx_filehandle_t fd, uint64_t length) {
#endif
}
int mdbx_fseek(mdbx_filehandle_t fd, uint64_t pos) {
#if defined(_WIN32) || defined(_WIN64)
LARGE_INTEGER li;
li.QuadPart = pos;
return SetFilePointerEx(fd, li, NULL, FILE_BEGIN) ? MDBX_SUCCESS
: GetLastError();
#else
STATIC_ASSERT_MSG(sizeof(off_t) >= sizeof(size_t),
"libmdbx requires 64-bit file I/O on 64-bit systems");
return (lseek(fd, pos, SEEK_SET) < 0) ? errno : MDBX_SUCCESS;
#endif
}
/*----------------------------------------------------------------------------*/
int mdbx_thread_create(mdbx_thread_t *thread,
@@ -973,11 +1001,11 @@ int mdbx_mresize(int flags, mdbx_mmap_t *map, size_t size, size_t limit) {
&ReservedSize, MEM_RESERVE, PAGE_NOACCESS);
if (!NT_SUCCESS(status)) {
ReservedAddress = NULL;
if (status != /* STATUS_CONFLICTING_ADDRESSES */ 0xC0000018 ||
limit == map->length)
if (status != /* STATUS_CONFLICTING_ADDRESSES */ 0xC0000018)
goto bailout_ntstatus /* no way to recovery */;
/* assume we can change base address if mapping size changed */
/* assume we can change base address if mapping size changed or prev address
* couldn't be used */
map->address = NULL;
}
@@ -1034,8 +1062,8 @@ retry_mapview:;
if (!NT_SUCCESS(status)) {
if (status == /* STATUS_CONFLICTING_ADDRESSES */ 0xC0000018 &&
map->address && limit != map->length) {
/* try remap at another base address, but only if the limit is changing */
map->address) {
/* try remap at another base address */
map->address = NULL;
goto retry_mapview;
}

View File

@@ -1,7 +1,7 @@
/* https://en.wikipedia.org/wiki/Operating_system_abstraction_layer */
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -473,10 +473,12 @@ int mdbx_thread_join(mdbx_thread_t thread);
int mdbx_filesync(mdbx_filehandle_t fd, bool fullsync);
int mdbx_filesize_sync(mdbx_filehandle_t fd);
int mdbx_ftruncate(mdbx_filehandle_t fd, uint64_t length);
int mdbx_fseek(mdbx_filehandle_t fd, uint64_t pos);
int mdbx_filesize(mdbx_filehandle_t fd, uint64_t *length);
int mdbx_openfile(const char *pathname, int flags, mode_t mode,
mdbx_filehandle_t *fd);
int mdbx_closefile(mdbx_filehandle_t fd);
int mdbx_removefile(const char *pathname);
typedef struct mdbx_mmap_param {
union {
@@ -548,7 +550,6 @@ int mdbx_lck_init(MDBX_env *env);
int mdbx_lck_seize(MDBX_env *env);
int mdbx_lck_downgrade(MDBX_env *env, bool complete);
int mdbx_lck_upgrade(MDBX_env *env);
void mdbx_lck_destroy(MDBX_env *env);
int mdbx_rdt_lock(MDBX_env *env);

View File

@@ -1,7 +1,7 @@
/* mdbx_chk.c - memory-mapped database check tool */
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -61,12 +61,18 @@ static void signal_handler(int sig) {
#define EXIT_FAILURE_CHECK_MAJOR (EXIT_FAILURE + 1)
#define EXIT_FAILURE_CHECK_MINOR EXIT_FAILURE
typedef struct {
const char *name;
struct {
uint64_t total;
uint64_t empty;
} pages;
uint64_t payload_bytes;
uint64_t lost_bytes;
} walk_dbi_t;
struct {
const char *dbi_names[MAX_DBI];
uint64_t dbi_pages[MAX_DBI];
uint64_t dbi_empty_pages[MAX_DBI];
uint64_t dbi_payload_bytes[MAX_DBI];
uint64_t dbi_lost_bytes[MAX_DBI];
walk_dbi_t dbi[MAX_DBI];
short *pagemap;
uint64_t total_payload_bytes;
uint64_t pgcount;
@@ -95,7 +101,7 @@ struct problem *problems_list;
uint64_t total_problems;
static void
#ifdef __GNU__
#ifdef __GNUC__
__attribute__((format(printf, 1, 2)))
#endif
print(const char *msg, ...) {
@@ -110,7 +116,7 @@ static void
}
static void
#ifdef __GNU__
#ifdef __GNUC__
__attribute__((format(printf, 1, 2)))
#endif
error(const char *msg, ...) {
@@ -131,9 +137,9 @@ static void pagemap_cleanup(void) {
int i;
for (i = 1; i < MAX_DBI; ++i) {
if (walk.dbi_names[i]) {
free((void *)walk.dbi_names[i]);
walk.dbi_names[i] = NULL;
if (walk.dbi[i].name) {
free((void *)walk.dbi[i].name);
walk.dbi[i].name = NULL;
}
}
@@ -141,32 +147,35 @@ static void pagemap_cleanup(void) {
walk.pagemap = NULL;
}
static int pagemap_lookup_dbi(const char *dbi) {
static int last;
int i;
static walk_dbi_t *pagemap_lookup_dbi(const char *dbi_name) {
static walk_dbi_t *last;
if (last > 0 && strcmp(walk.dbi_names[last], dbi) == 0)
if (last && strcmp(last->name, dbi_name) == 0)
return last;
for (i = 1; walk.dbi_names[i] && last < MAX_DBI; ++i)
if (strcmp(walk.dbi_names[i], dbi) == 0)
return last = i;
if (i == MAX_DBI)
return -1;
walk.dbi_names[i] = strdup(dbi);
walk_dbi_t *dbi = walk.dbi + 1;
while (dbi->name) {
if (strcmp(dbi->name, dbi_name) == 0)
return last = dbi;
if (++dbi == walk.dbi + MAX_DBI)
return NULL;
}
dbi->name = strdup(dbi_name);
if (verbose > 1) {
print(" - found '%s' area\n", dbi);
print(" - found '%s' area\n", dbi_name);
fflush(NULL);
}
return last = i;
return last = dbi;
}
static void problem_add(const char *object, uint64_t entry_number,
const char *msg, const char *extra, ...) {
static void
#ifdef __GNUC__
__attribute__((format(printf, 4, 5)))
#endif
problem_add(const char *object, uint64_t entry_number, const char *msg,
const char *extra, ...) {
total_problems++;
if (!quiet) {
@@ -233,7 +242,7 @@ static uint64_t problems_pop(struct problem *list) {
}
static int pgvisitor(uint64_t pgno, unsigned pgnumber, void *ctx,
const char *dbi, const char *type, size_t nentries,
const char *dbi_name, const char *type, size_t nentries,
size_t payload_bytes, size_t header_bytes,
size_t unused_bytes) {
(void)ctx;
@@ -241,54 +250,58 @@ static int pgvisitor(uint64_t pgno, unsigned pgnumber, void *ctx,
if (type) {
uint64_t page_bytes = payload_bytes + header_bytes + unused_bytes;
size_t page_size = (size_t)pgnumber * envstat.ms_psize;
int index = pagemap_lookup_dbi(dbi);
if (index < 0)
walk_dbi_t *dbi = pagemap_lookup_dbi(dbi_name);
if (!dbi)
return MDBX_ENOMEM;
if (verbose > 2 && (!only_subdb || strcmp(only_subdb, dbi) == 0)) {
if (verbose > 2 && (!only_subdb || strcmp(only_subdb, dbi_name) == 0)) {
if (pgnumber == 1)
print(" %s-page %" PRIu64, type, pgno);
else
print(" %s-span %" PRIu64 "[%u]", type, pgno, pgnumber);
print(" of %s: header %" PRIiPTR ", payload %" PRIiPTR
", unused %" PRIiPTR "\n",
dbi, header_bytes, payload_bytes, unused_bytes);
dbi_name, header_bytes, payload_bytes, unused_bytes);
}
walk.pgcount += pgnumber;
if (unused_bytes > page_size)
problem_add("page", pgno, "illegal unused-bytes", "%u < %i < %u", 0,
unused_bytes, envstat.ms_psize);
problem_add("page", pgno, "illegal unused-bytes",
"%u < %" PRIuPTR " < %u", 0, unused_bytes, envstat.ms_psize);
if (header_bytes < (int)sizeof(long) ||
(size_t)header_bytes >= envstat.ms_psize - sizeof(long))
problem_add("page", pgno, "illegal header-length",
"%" PRIuPTR " < %i < %" PRIuPTR "", sizeof(long),
"%" PRIuPTR " < %" PRIuPTR " < %" PRIuPTR, sizeof(long),
header_bytes, envstat.ms_psize - sizeof(long));
if (payload_bytes < 1) {
if (nentries > 1) {
problem_add("page", pgno, "zero size-of-entry",
"payload %i bytes, %i entries", payload_bytes, nentries);
"payload %" PRIuPTR " bytes, %" PRIuPTR " entries",
payload_bytes, nentries);
if ((size_t)header_bytes + unused_bytes < page_size) {
/* LY: hush a misuse error */
page_bytes = page_size;
}
} else {
problem_add("page", pgno, "empty", "payload %i bytes, %i entries",
problem_add("page", pgno, "empty",
"payload %" PRIuPTR " bytes, %" PRIuPTR " entries",
payload_bytes, nentries);
walk.dbi_empty_pages[index] += 1;
dbi->pages.empty += 1;
}
}
if (page_bytes != page_size) {
problem_add("page", pgno, "misused",
"%" PRIu64 " != %" PRIu64 " (%ih + %ip + %iu)", page_size,
page_bytes, header_bytes, payload_bytes, unused_bytes);
"%" PRIu64 " != %" PRIu64 " (%" PRIuPTR "h + %" PRIuPTR
"p + %" PRIuPTR "u)",
page_size, page_bytes, header_bytes, payload_bytes,
unused_bytes);
if (page_size > page_bytes)
walk.dbi_lost_bytes[index] += page_size - page_bytes;
dbi->lost_bytes += page_size - page_bytes;
} else {
walk.dbi_payload_bytes[index] += payload_bytes + header_bytes;
dbi->payload_bytes += payload_bytes + header_bytes;
walk.total_payload_bytes += payload_bytes + header_bytes;
}
@@ -299,10 +312,10 @@ static int pgvisitor(uint64_t pgno, unsigned pgnumber, void *ctx,
"%" PRIu64 " > %" PRIu64 "", pgno, lastpgno);
else if (walk.pagemap[pgno])
problem_add("page", pgno, "already used", "in %s",
walk.dbi_names[walk.pagemap[pgno]]);
walk.dbi[walk.pagemap[pgno]].name);
else {
walk.pagemap[pgno] = (short)index;
walk.dbi_pages[index] += 1;
walk.pagemap[pgno] = (short)(dbi - walk.dbi);
dbi->pages.total += 1;
}
++pgno;
} while (--pgnumber);
@@ -337,16 +350,22 @@ static int handle_freedb(const uint64_t record_number, const MDBX_val *key,
problem_add("entry", record_number, "wrong txn-id", "%" PRIaTXN "", txnid);
if (data->iov_len < sizeof(pgno_t) || data->iov_len % sizeof(pgno_t))
problem_add("entry", record_number, "wrong idl size", "%" PRIuPTR "",
problem_add("entry", txnid, "wrong idl size", "%" PRIuPTR "",
data->iov_len);
else {
const pgno_t number = *iptr++;
if (number >= MDBX_PNL_UM_MAX)
problem_add("entry", record_number, "wrong idl length", "%" PRIiPTR "",
number);
else if ((number + 1) * sizeof(pgno_t) != data->iov_len)
problem_add("entry", record_number, "mismatch idl length",
"%" PRIuSIZE " != %" PRIuSIZE "",
if (number < 1 || number >= INT_MAX / 2)
problem_add("entry", txnid, "wrong idl length", "%" PRIaPGNO, number);
else if ((number + 1) * sizeof(pgno_t) > data->iov_len)
problem_add("entry", txnid, "trimmed idl",
"%" PRIuSIZE " > %" PRIuSIZE " (corruption)",
(number + 1) * sizeof(pgno_t), data->iov_len);
else if (data->iov_len - (number + 1) * sizeof(pgno_t) >=
/* LY: allow gap upto one page. it is ok
* and better than shink-and-retry inside mdbx_update_gc() */
envstat.ms_psize)
problem_add("entry", txnid, "extra idl space",
"%" PRIuSIZE " < %" PRIuSIZE " (minor, not a trouble)",
(number + 1) * sizeof(pgno_t), data->iov_len);
else {
freedb_pages += number;
@@ -359,12 +378,12 @@ static int handle_freedb(const uint64_t record_number, const MDBX_val *key,
for (unsigned i = 0; i < number; ++i) {
const pgno_t pg = iptr[i];
if (pg < NUM_METAS || pg > envinfo.mi_last_pgno)
problem_add("entry", record_number, "wrong idl entry",
problem_add("entry", txnid, "wrong idl entry",
"%u < %" PRIaPGNO " < %" PRIu64 "", NUM_METAS, pg,
envinfo.mi_last_pgno);
else if (MDBX_PNL_DISORDERED(prev, pg)) {
bad = " [bad sequence]";
problem_add("entry", record_number, "bad sequence",
problem_add("entry", txnid, "bad sequence",
"%" PRIaPGNO " <> %" PRIaPGNO "", prev, pg);
}
prev = pg;
@@ -516,7 +535,7 @@ static int process_db(MDBX_dbi dbi, char *name, visitor *handler, bool silent) {
if (key.iov_len > maxkeysize) {
problem_add("entry", record_count, "key length exceeds max-key-size",
"%" PRIuPTR " > %u", key.iov_len, maxkeysize);
"%" PRIuPTR " > %" PRIuPTR, key.iov_len, maxkeysize);
} else if ((flags & MDBX_INTEGERKEY) && key.iov_len != sizeof(uint64_t) &&
key.iov_len != sizeof(uint32_t)) {
problem_add("entry", record_count, "wrong key length",
@@ -758,7 +777,7 @@ static void print_size(const char *prefix, const uint64_t value,
}
int main(int argc, char *argv[]) {
int i, rc;
int rc;
char *prog = argv[0];
char *envname;
int problems_maindb = 0, problems_freedb = 0, problems_meta = 0;
@@ -778,14 +797,14 @@ int main(int argc, char *argv[]) {
}
#endif
walk.dbi_names[0] = "@gc";
walk.dbi[FREE_DBI].name = "@gc";
atexit(pagemap_cleanup);
if (argc < 2) {
usage(prog);
}
while ((i = getopt(argc, argv, "Vvqnwcds:")) != EOF) {
for (int i; (i = getopt(argc, argv, "Vvqnwcds:")) != EOF;) {
switch (i) {
case 'V':
printf("%s (%s, build %s)\n", mdbx_version.git.describe,
@@ -988,24 +1007,25 @@ int main(int argc, char *argv[]) {
goto bailout;
}
uint64_t n;
for (n = 0; n < lastpgno; ++n)
for (uint64_t n = 0; n < lastpgno; ++n)
if (!walk.pagemap[n])
walk.dbi_pages[0] += 1;
walk.dbi[FREE_DBI].pages.total += 1;
empty_pages = lost_bytes = 0;
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
empty_pages += walk.dbi_empty_pages[i];
lost_bytes += walk.dbi_lost_bytes[i];
for (walk_dbi_t *dbi = walk.dbi; ++dbi < walk.dbi + MAX_DBI && dbi->name;) {
empty_pages += dbi->pages.empty;
lost_bytes += dbi->lost_bytes;
}
if (verbose) {
uint64_t total_page_bytes = walk.pgcount * envstat.ms_psize;
print(" - dbi pages: %" PRIu64 " total", walk.pgcount);
if (verbose > 1)
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i)
print(", %s %" PRIu64 "", walk.dbi_names[i], walk.dbi_pages[i]);
print(", %s %" PRIu64 "\n", walk.dbi_names[0], walk.dbi_pages[0]);
for (walk_dbi_t *dbi = walk.dbi;
++dbi < walk.dbi + MAX_DBI && dbi->name;)
print(", %s %" PRIu64, dbi->name, dbi->pages.total);
print(", %s %" PRIu64 "\n", walk.dbi[FREE_DBI].name,
walk.dbi[FREE_DBI].pages.total);
if (verbose > 1) {
print(" - space info: total %" PRIu64 " bytes, payload %" PRIu64
" (%.1f%%), unused "
@@ -1015,19 +1035,19 @@ int main(int argc, char *argv[]) {
total_page_bytes - walk.total_payload_bytes,
(total_page_bytes - walk.total_payload_bytes) * 100.0 /
total_page_bytes);
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
uint64_t dbi_bytes = walk.dbi_pages[i] * envstat.ms_psize;
for (walk_dbi_t *dbi = walk.dbi;
++dbi < walk.dbi + MAX_DBI && dbi->name;) {
uint64_t dbi_bytes = dbi->pages.total * envstat.ms_psize;
print(" %s: subtotal %" PRIu64 " bytes (%.1f%%),"
" payload %" PRIu64 " (%.1f%%), unused %" PRIu64 " (%.1f%%)",
walk.dbi_names[i], dbi_bytes,
dbi_bytes * 100.0 / total_page_bytes, walk.dbi_payload_bytes[i],
walk.dbi_payload_bytes[i] * 100.0 / dbi_bytes,
dbi_bytes - walk.dbi_payload_bytes[i],
(dbi_bytes - walk.dbi_payload_bytes[i]) * 100.0 / dbi_bytes);
if (walk.dbi_empty_pages[i])
print(", %" PRIu64 " empty pages", walk.dbi_empty_pages[i]);
if (walk.dbi_lost_bytes[i])
print(", %" PRIu64 " bytes lost", walk.dbi_lost_bytes[i]);
dbi->name, dbi_bytes, dbi_bytes * 100.0 / total_page_bytes,
dbi->payload_bytes, dbi->payload_bytes * 100.0 / dbi_bytes,
dbi_bytes - dbi->payload_bytes,
(dbi_bytes - dbi->payload_bytes) * 100.0 / dbi_bytes);
if (dbi->pages.empty)
print(", %" PRIu64 " empty pages", dbi->pages.empty);
if (dbi->lost_bytes)
print(", %" PRIu64 " bytes lost", dbi->lost_bytes);
print("\n");
}
}
@@ -1084,9 +1104,9 @@ int main(int argc, char *argv[]) {
error("used pages mismatch (%" PRIu64 " != %" PRIu64 ")\n",
walk.pgcount, lastpgno - freedb_pages);
}
if (walk.dbi_pages[0] != freedb_pages) {
if (walk.dbi[FREE_DBI].pages.total != freedb_pages) {
error("gc pages mismatch (%" PRIu64 " != %" PRIu64 ")\n",
walk.dbi_pages[0], freedb_pages);
walk.dbi[FREE_DBI].pages.total, freedb_pages);
}
} else if (verbose) {
print(" - skip check used and gc pages (btree-traversal with "

View File

@@ -1,4 +1,4 @@
.\" Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>.
.\" Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>.
.\" Copyright 2012-2015 Howard Chu, Symas Corp. All Rights Reserved.
.\" Copyright 2015,2016 Peter-Service R&D LLC <http://billing.ru/>.
.\" Copying restrictions apply. See COPYRIGHT/LICENSE.

View File

@@ -1,7 +1,7 @@
/* mdbx_copy.c - memory-mapped database backup tool */
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

View File

@@ -1,4 +1,4 @@
.\" Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>.
.\" Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>.
.\" Copyright 2014-2015 Howard Chu, Symas Corp. All Rights Reserved.
.\" Copyright 2015,2016 Peter-Service R&D LLC <http://billing.ru/>.
.\" Copying restrictions apply. See COPYRIGHT/LICENSE.

View File

@@ -1,7 +1,7 @@
/* mdbx_dump.c - memory-mapped database dump tool */
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

View File

@@ -1,4 +1,4 @@
.\" Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>.
.\" Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>.
.\" Copyright 2014-2015 Howard Chu, Symas Corp. All Rights Reserved.
.\" Copyright 2015,2016 Peter-Service R&D LLC <http://billing.ru/>.
.\" Copying restrictions apply. See COPYRIGHT/LICENSE.

View File

@@ -1,7 +1,7 @@
/* mdbx_load.c - memory-mapped database load tool */
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

View File

@@ -1,4 +1,4 @@
.\" Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>.
.\" Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>.
.\" Copyright 2012-2015 Howard Chu, Symas Corp. All Rights Reserved.
.\" Copyright 2015,2016 Peter-Service R&D LLC <http://billing.ru/>.
.\" Copying restrictions apply. See COPYRIGHT/LICENSE.

View File

@@ -1,7 +1,7 @@
/* mdbx_stat.c - memory-mapped database status tool */
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -18,8 +18,8 @@
#error "API version mismatch!"
#endif
#define MDBX_VERSION_RELEASE 5
#define MDBX_VERSION_REVISION 1
#define MDBX_VERSION_RELEASE 7
#define MDBX_VERSION_REVISION 0
/*LIBMDBX_EXPORTS*/ const mdbx_version_info mdbx_version = {
MDBX_VERSION_MAJOR,
@@ -30,5 +30,5 @@
"@MDBX_GIT_DESCRIBE@"}};
/*LIBMDBX_EXPORTS*/ const mdbx_build_info mdbx_build = {
"@MDBX_BUILD_TIMESTAMP@", "@MDBX_BUILD_TAGRET@", "@MDBX_BUILD_OPTIONS@",
"@MDBX_BUILD_TIMESTAMP@", "@MDBX_BUILD_TARGET@", "@MDBX_BUILD_OPTIONS@",
"@MDBX_BUILD_COMPILER@", "@MDBX_BUILD_FLAGS@"};

View File

@@ -5,30 +5,31 @@ set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-missing-declarations")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-cast-qual")
add_executable(${TARGET}
base.h
cases.cc
chrono.cc
chrono.h
config.cc
config.h
dead.cc
hill.cc
jitter.cc
keygen.cc
keygen.h
log.cc
log.h
main.cc
osal.h
osal-unix.cc
test.cc
test.h
try.cc
utils.cc
utils.h
)
base.h
cases.cc
chrono.cc
chrono.h
config.cc
config.h
copy.cc
dead.cc
hill.cc
jitter.cc
keygen.cc
keygen.h
log.cc
log.h
main.cc
osal.h
osal-unix.cc
test.cc
test.h
try.cc
utils.cc
utils.h
)
target_link_libraries(${TARGET}
mdbx
)
mdbx
)

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -68,6 +68,7 @@ void testcase_setup(const char *casename, actor_params &params,
configure_actor(last_space_id, ac_jitter, nullptr, params);
configure_actor(last_space_id, ac_hill, nullptr, params);
configure_actor(last_space_id, ac_try, nullptr, params);
configure_actor(last_space_id, ac_copy, nullptr, params);
log_notice("<<< testcase_setup(%s): done", casename);
} else {
failure("unknown testcase `%s`", casename);

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -43,6 +43,11 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
if (narg + 1 < argc && strncmp("--", argv[narg + 1], 2) != 0) {
*value = argv[narg + 1];
if (strcmp(*value, "default") == 0) {
if (!default_value)
failure("Option '--%s' doen't accept default value\n", option);
*value = default_value;
}
++narg;
return true;
}
@@ -57,9 +62,15 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty) {
return parse_option(argc, argv, narg, option, value, allow_empty,
allow_empty ? "" : nullptr);
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty,
const char *default_value) {
const char *value_cstr;
if (!parse_option(argc, argv, narg, option, &value_cstr,
allow_empty ? "" : nullptr))
if (!parse_option(argc, argv, narg, option, &value_cstr, default_value))
return false;
if (!allow_empty && strlen(value_cstr) == 0)
@@ -75,7 +86,7 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
if (!parse_option(argc, argv, narg, option, &list))
return false;
mask = 0;
unsigned clear = 0;
while (*list) {
if (*list == ',' || *list == ' ' || *list == '\t') {
++list;
@@ -83,14 +94,21 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
}
const char *const comma = strchr(list, ',');
const bool strikethrough = *list == '-' || *list == '~';
if (strikethrough || *list == '+')
++list;
else
mask = clear;
const size_t len = (comma) ? comma - list : strlen(list);
const option_verb *scan = verbs;
while (true) {
if (!scan->verb)
failure("Unknown verb '%.*s', for option '==%s'\n", (int)len, list,
option);
if (strlen(scan->verb) == len && strncmp(list, scan->verb, len) == 0) {
mask |= scan->mask;
mask = strikethrough ? mask & ~scan->mask : mask | scan->mask;
clear = strikethrough ? clear & ~scan->mask : clear | scan->mask;
list += len;
break;
}
@@ -103,15 +121,36 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint64_t &value, const scale_mode scale,
const uint64_t minval, const uint64_t maxval) {
const uint64_t minval, const uint64_t maxval,
const uint64_t default_value) {
const char *value_cstr;
if (!parse_option(argc, argv, narg, option, &value_cstr))
return false;
if (default_value && strcmp(value_cstr, "default") == 0) {
value = default_value;
return true;
}
if (strcmp(value_cstr, "min") == 0 || strcmp(value_cstr, "minimal") == 0) {
value = minval;
return true;
}
if (strcmp(value_cstr, "max") == 0 || strcmp(value_cstr, "maximal") == 0) {
value = maxval;
return true;
}
char *suffix = nullptr;
errno = 0;
unsigned long raw = strtoul(value_cstr, &suffix, 0);
unsigned long long raw = strtoull(value_cstr, &suffix, 0);
if ((suffix && *suffix) || errno) {
suffix = nullptr;
errno = 0;
raw = strtoull(value_cstr, &suffix, 10);
}
if (errno)
failure("Option '--%s' expects a numeric value (%s)\n", option,
test_strerror(errno));
@@ -167,28 +206,58 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
unsigned &value, const scale_mode scale,
const unsigned minval, const unsigned maxval) {
const unsigned minval, const unsigned maxval,
const unsigned default_value) {
uint64_t huge;
if (!parse_option(argc, argv, narg, option, huge, scale, minval, maxval))
if (!parse_option(argc, argv, narg, option, huge, scale, minval, maxval,
default_value))
return false;
value = (unsigned)huge;
return true;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint8_t &value, const uint8_t minval, const uint8_t maxval) {
uint8_t &value, const uint8_t minval, const uint8_t maxval,
const uint8_t default_value) {
uint64_t huge;
if (!parse_option(argc, argv, narg, option, huge, no_scale, minval, maxval))
if (!parse_option(argc, argv, narg, option, huge, no_scale, minval, maxval,
default_value))
return false;
value = (uint8_t)huge;
return true;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int64_t &value, const int64_t minval, const int64_t maxval,
const int64_t default_value) {
uint64_t proxy = (uint64_t)value;
if (parse_option(argc, argv, narg, option, proxy, config::binary,
(uint64_t)minval, (uint64_t)maxval,
(uint64_t)default_value)) {
value = (int64_t)proxy;
return true;
}
return false;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int32_t &value, const int32_t minval, const int32_t maxval,
const int32_t default_value) {
uint64_t proxy = (uint64_t)value;
if (parse_option(argc, argv, narg, option, proxy, config::binary,
(uint64_t)minval, (uint64_t)maxval,
(uint64_t)default_value)) {
value = (int32_t)proxy;
return true;
}
return false;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool &value) {
const char *value_cstr = NULL;
const char *value_cstr = nullptr;
if (!parse_option(argc, argv, narg, option, &value_cstr, "yes")) {
const char *current = argv[narg];
if (strncmp(current, "--no-", 5) == 0 && strcmp(current + 5, option) == 0) {
@@ -257,7 +326,7 @@ static void dump_verbs(const char *caption, size_t bits,
++verbs;
}
logging::feed("\n");
logging::feed("%s\n", (*comma == '\0') ? "none" : "");
}
static void dump_duration(const char *caption, unsigned duration) {
@@ -288,8 +357,12 @@ void dump(const char *title) {
: i->params.pathname_log.c_str());
}
log_info("database: %s, size %" PRIu64 "\n", i->params.pathname_db.c_str(),
i->params.size);
log_info("database: %s, size %" PRIuPTR "[%" PRIiPTR "..%" PRIiPTR
", %i %i, %i]\n",
i->params.pathname_db.c_str(), i->params.size_now,
i->params.size_lower, i->params.size_upper,
i->params.shrink_threshold, i->params.growth_step,
i->params.pagesize);
dump_verbs("mode", i->params.mode_flags, mode_bits);
dump_verbs("table", i->params.table_flags, table_bits);
@@ -306,7 +379,13 @@ void dump(const char *title) {
log_info("threads %u\n", i->params.nthreads);
log_info("keygen.case: %s\n", keygencase2str(i->params.keygen.keycase));
log_info(
"keygen.params: case %s, width %u, mesh %u, rotate %u, offset %" PRIu64
", split %u/%u\n",
keygencase2str(i->params.keygen.keycase), i->params.keygen.width,
i->params.keygen.mesh, i->params.keygen.rotate, i->params.keygen.offset,
i->params.keygen.split,
i->params.keygen.width - i->params.keygen.split);
log_info("keygen.seed: %u\n", i->params.keygen.seed);
log_info("key: minlen %u, maxlen %u\n", i->params.keylen_min,
i->params.keylen_max);
@@ -469,3 +548,27 @@ bool actor_config::deserialize(const char *str, actor_config &config) {
TRACE("<< actor_config::deserialize: OK\n");
return true;
}
unsigned actor_params::mdbx_keylen_min() const {
return (table_flags & MDBX_INTEGERKEY) ? 4 : 0;
}
unsigned actor_params::mdbx_keylen_max() const {
return (table_flags & MDBX_INTEGERKEY)
? 8
: std::min((unsigned)mdbx_get_maxkeysize(pagesize),
(unsigned)UINT16_MAX);
}
unsigned actor_params::mdbx_datalen_min() const {
return (table_flags & MDBX_INTEGERDUP) ? 4 : 0;
}
unsigned actor_params::mdbx_datalen_max() const {
return (table_flags & MDBX_INTEGERDUP)
? 8
: std::min((table_flags & MDBX_DUPSORT)
? (unsigned)mdbx_get_maxkeysize(pagesize)
: (unsigned)MDBX_MAXDATASIZE,
(unsigned)UINT16_MAX);
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -26,7 +26,8 @@ enum actor_testcase {
ac_deadread,
ac_deadwrite,
ac_jitter,
ac_try
ac_try,
ac_copy
};
enum actor_status {
@@ -62,6 +63,10 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty = false);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty,
const char *default_value);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool &value);
@@ -75,16 +80,25 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint64_t &value, const scale_mode scale,
const uint64_t minval = 0, const uint64_t maxval = INT64_MAX);
const uint64_t minval = 0, const uint64_t maxval = INT64_MAX,
const uint64_t default_value = 0);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
unsigned &value, const scale_mode scale,
const unsigned minval = 0, const unsigned maxval = INT32_MAX);
const unsigned minval = 0, const unsigned maxval = INT32_MAX,
const unsigned default_value = 0);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint8_t &value, const uint8_t minval = 0,
const uint8_t maxval = 255);
const uint8_t maxval = 255, const uint8_t default_value = 0);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int64_t &value, const int64_t minval, const int64_t maxval,
const int64_t default_value = -1);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int32_t &value, const int32_t minval, const int32_t maxval,
const int32_t default_value = -1);
//-----------------------------------------------------------------------------
#pragma pack(push, 1)
@@ -121,6 +135,8 @@ struct keygen_params_pod {
* Иначе говоря, нет смысла в со-координации генерации паттернов для
* ключей и значений. Более того, генерацию значений всегда необходимо
* рассматривать в контексте связки с одним значением ключа.
* - Тем не менее, во всех случаях достаточно важным является равномерная
* всех возможных сочетаний длин ключей и данных.
*
* width:
* Большинство тестов предполагают создание или итерирование некоторого
@@ -156,7 +172,7 @@ struct keygen_params_pod {
* псевдо-случайные значений ключей без псевдо-случайности в значениях.
*
* Такое ограничение соответствуют внутренней алгоритмике libmdbx. Проще
* говоря мы можем проверить движок псевдо-случайной последовательностью
* говоря, мы можем проверить движок псевдо-случайной последовательностью
* ключей на таблицах без дубликатов (без multi-value), а затем проверить
* корректность работу псевдо-случайной последовательностью значений на
* таблицах с дубликатами (с multi-value), опционально добавляя
@@ -203,7 +219,12 @@ struct actor_params_pod {
unsigned mode_flags;
unsigned table_flags;
uint64_t size;
intptr_t size_lower;
intptr_t size_now;
intptr_t size_upper;
int shrink_threshold;
int growth_step;
int pagesize;
unsigned test_duration;
unsigned test_nops;
@@ -246,6 +267,11 @@ struct actor_params : public config::actor_params_pod {
std::string pathname_log;
std::string pathname_db;
void set_defaults(const std::string &tmpdir);
unsigned mdbx_keylen_min() const;
unsigned mdbx_keylen_max() const;
unsigned mdbx_datalen_min() const;
unsigned mdbx_datalen_max() const;
};
struct actor_config : public config::actor_config_pod {

26
test/copy.cc Normal file
View File

@@ -0,0 +1,26 @@
#include "test.h"
void testcase_copy::copy_db(const bool with_compaction) {
int err = osal_removefile(copy_pathname);
if (err != MDBX_SUCCESS && err != MDBX_ENOFILE)
failure_perror("mdbx_removefile()", err);
err = mdbx_env_copy(db_guard.get(), copy_pathname.c_str(),
with_compaction ? MDBX_CP_COMPACT : 0);
if (unlikely(err != MDBX_SUCCESS))
failure_perror(with_compaction ? "mdbx_env_copy(MDBX_CP_COMPACT)"
: "mdbx_env_copy(MDBX_CP_ASIS)",
err);
}
bool testcase_copy::run() {
jitter_delay();
db_open();
assert(!txn_guard);
const bool order = flipcoin();
jitter_delay();
copy_db(order);
jitter_delay();
copy_db(!order);
return true;
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

66
test/gc.sh Executable file
View File

@@ -0,0 +1,66 @@
#!/bin/bash
set -euo pipefail
make check
TESTDB_PREFIX=${1:-/dev/shm/mdbx-gc-test}.
function rep9 { printf "%*s" $1 '' | tr ' ' '9'; }
function join { local IFS="$1"; shift; echo "$*"; }
function bit2option { local -n arr=$1; (( ($2&(1<<$3)) != 0 )) && echo -n '+' || echo -n '-'; echo "${arr[$3]}"; }
options=(writemap coalesce lifo)
function bits2list {
local -n arr=$1
local i
local list=()
for ((i=0; i<${#arr[@]}; ++i)) do
list[$i]=$(bit2option $1 $2 $i)
done
join , "${list[@]}"
}
function probe {
echo "=============================================== $(date)"
echo "${caption}: $*"
rm -f ${TESTDB_PREFIX}* \
&& ./mdbx_test --pathname=${TESTDB_PREFIX}db "$@" | lz4 > ${TESTDB_PREFIX}log.lz4 \
&& ./mdbx_chk -nvv ${TESTDB_PREFIX}db | tee ${TESTDB_PREFIX}chk \
&& ./mdbx_chk -nvv ${TESTDB_PREFIX}db-copy | tee ${TESTDB_PREFIX}chk-copy \
|| (echo "FAILED"; exit 1)
}
###############################################################################
count=0
for nops in {2..7}; do
for ((wbatch=nops-1; wbatch > 0; --wbatch)); do
loops=$(((3333 >> nops) / nops + 1))
for ((rep=0; rep++ < loops; )); do
for ((bits=2**${#options[@]}; --bits >= 0; )); do
seed=$(date +%N)
caption="Probe #$((++count)) int-key,w/o-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+key.integer,-data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) int-key,with-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+key.integer,+data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) int-key,int-data, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+key.integer,+data.integer --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) w/o-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=-data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) with-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
done
done
done
done
echo "=== ALL DONE ====================== $(date)"

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -53,7 +53,7 @@ bool testcase_hill::run() {
*/
/* TODO: работа в несколько потоков */
keyvalue_maker.setup(config.params, 0 /* thread_number */);
keyvalue_maker.setup(config.params, config.actor_id, 0 /* thread_number */);
keygen::buffer a_key = keygen::alloc(config.params.keylen_max);
keygen::buffer a_data_0 = keygen::alloc(config.params.datalen_max);
@@ -65,7 +65,9 @@ bool testcase_hill::run() {
? MDBX_NODUPDATA
: MDBX_NODUPDATA | MDBX_NOOVERWRITE;
const unsigned update_flags =
MDBX_CURRENT | MDBX_NODUPDATA | MDBX_NOOVERWRITE;
(config.params.table_flags & MDBX_DUPSORT)
? MDBX_CURRENT | MDBX_NODUPDATA | MDBX_NOOVERWRITE
: MDBX_NODUPDATA;
uint64_t serial_count = 0;
unsigned txn_nops = 0;
@@ -115,7 +117,7 @@ bool testcase_hill::run() {
rc = mdbx_replace(txn_guard.get(), dbi, &a_key->value, &a_data_0->value,
&a_data_1->value, update_flags);
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_put(update-a: 1->0)", rc);
failure_perror("mdbx_replace(update-a: 1->0)", rc);
if (++txn_nops >= config.params.batch_write) {
txn_restart(false, false);
@@ -156,8 +158,6 @@ bool testcase_hill::run() {
a_serial);
generate_pair(a_serial, a_key, a_data_0, 0);
generate_pair(a_serial, a_key, a_data_1, age_shift);
if (a_serial == 808)
log_trace("!!!");
int rc = mdbx_replace(txn_guard.get(), dbi, &a_key->value, &a_data_1->value,
&a_data_0->value, update_flags);
if (unlikely(rc != MDBX_SUCCESS))

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -58,7 +58,11 @@ bool testcase_jitter::run() {
jitter_delay();
db_close();
report(1);
/* just 'align' nops with other tests with batching */
const auto batching =
std::max(config.params.batch_read, config.params.batch_write);
report(std::max(1u, batching / 2));
}
return true;
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -30,7 +30,7 @@ serial_t injective(const serial_t serial,
/* LY: All these "magic" prime numbers were found
* and verified with a bit of brute force. */
static const uint64_t m[64 - serial_minwith] = {
static const uint64_t m[64 - serial_minwith + 1] = {
/* 8 - 24 */
113, 157, 397, 653, 1753, 5641, 9697, 23873, 25693, 80833, 105953, 316937,
309277, 834497, 1499933, 4373441, 10184137,
@@ -43,26 +43,31 @@ serial_t injective(const serial_t serial,
2420886491930041, 3601632139991929, 11984491914483833, 21805846439714153,
23171543400565993, 53353226456762893, 155627817337932409,
227827205384840249, 816509268558278821, 576933057762605689,
2623957345935638441, 5048241705479929949, 4634245581946485653};
static const uint8_t s[64 - serial_minwith] = {
2623957345935638441, 5048241705479929949, 4634245581946485653,
4613509448041658233, 4952535426879925961};
static const uint8_t s[64 - serial_minwith + 1] = {
/* 8 - 24 */
2, 3, 4, 4, 2, 4, 3, 3, 7, 3, 3, 4, 8, 3, 10, 3, 11,
/* 25 - 64 */
11, 9, 9, 9, 11, 10, 5, 14, 11, 16, 14, 12, 13, 16, 19, 10, 10, 21, 7, 20,
10, 14, 22, 19, 3, 21, 18, 19, 26, 24, 2, 21, 25, 29, 24, 10, 11, 14};
10, 14, 22, 19, 3, 21, 18, 19, 26, 24, 2, 21, 25, 29, 24, 10, 11, 14, 20,
19};
serial_t result = serial * m[bits - 8];
const auto mult = m[bits - 8];
const auto shift = s[bits - 8];
serial_t result = serial * mult;
if (salt) {
const unsigned left = bits / 2;
const unsigned right = bits - left;
result = (result << left) | ((result & mask(bits)) >> right);
result = (result ^ salt) * m[bits - 8];
result = (result ^ salt) * mult;
}
result ^= result << s[bits - 8];
result ^= result << shift;
result &= mask(bits);
log_trace("keygen-injective: serial %" PRIu64 " into %" PRIu64, serial,
result);
log_trace("keygen-injective: serial %" PRIu64 "/%u @%" PRIx64 ",%u,%" PRIu64
" => %" PRIu64 "/%u",
serial, bits, mult, shift, salt, result, bits);
return result;
}
@@ -73,8 +78,9 @@ void __hot maker::pair(serial_t serial, const buffer &key, buffer &value,
assert(mapping.mesh <= mapping.width);
assert(mapping.rotate <= mapping.width);
assert(mapping.offset <= mask(mapping.width));
assert(!(key_essentials.flags & (MDBX_INTEGERDUP | MDBX_REVERSEDUP)));
assert(!(value_essentials.flags & (MDBX_INTEGERKEY | MDBX_REVERSEKEY)));
assert(!(key_essentials.flags &
~(MDBX_INTEGERKEY | MDBX_REVERSEKEY | MDBX_DUPSORT)));
assert(!(value_essentials.flags & ~(MDBX_INTEGERDUP | MDBX_REVERSEDUP)));
log_trace("keygen-pair: serial %" PRIu64 ", data-age %" PRIu64, serial,
value_age);
@@ -82,31 +88,49 @@ void __hot maker::pair(serial_t serial, const buffer &key, buffer &value,
if (mapping.mesh >= serial_minwith) {
serial =
(serial & ~mask(mapping.mesh)) | injective(serial, mapping.mesh, salt);
log_trace("keygen-pair: mesh %" PRIu64, serial);
log_trace("keygen-pair: mesh@%u => %" PRIu64, mapping.mesh, serial);
}
if (mapping.rotate) {
const unsigned right = mapping.rotate;
const unsigned left = mapping.width - right;
serial = (serial << left) | ((serial & mask(mapping.width)) >> right);
log_trace("keygen-pair: rotate %" PRIu64 ", 0x%" PRIx64, serial, serial);
log_trace("keygen-pair: rotate@%u => %" PRIu64 ", 0x%" PRIx64,
mapping.rotate, serial, serial);
}
serial = (serial + mapping.offset) & mask(mapping.width);
log_trace("keygen-pair: offset %" PRIu64, serial);
serial += base;
if (mapping.offset) {
serial = (serial + mapping.offset) & mask(mapping.width);
log_trace("keygen-pair: offset@%" PRIu64 " => %" PRIu64, mapping.offset,
serial);
}
if (base) {
serial += base;
log_trace("keygen-pair: base@%" PRIu64 " => %" PRIu64, base, serial);
}
serial_t key_serial = serial;
serial_t value_serial = value_age;
serial_t value_serial = value_age << mapping.split;
if (mapping.split) {
key_serial = serial >> mapping.split;
value_serial =
(serial & mask(mapping.split)) | (value_age << mapping.split);
if (key_essentials.flags & MDBX_DUPSORT) {
key_serial >>= mapping.split;
value_serial += serial & mask(mapping.split);
} else {
/* Без MDBX_DUPSORT требуется уникальность ключей, а для этого нельзя
* отбрасывать какие-либо биты serial после инъективного преобразования.
* Поэтому key_serial не трогаем, а в value_serial нелинейно вмешиваем
* запрошенное количество бит из serial */
value_serial +=
(serial ^ (serial >> mapping.split)) & mask(mapping.split);
}
value_serial |= value_age << mapping.split;
log_trace("keygen-pair: split@%u => k%" PRIu64 ", v%" PRIu64, mapping.split,
key_serial, value_serial);
}
log_trace("keygen-pair: key %" PRIu64 ", value %" PRIu64, key_serial,
value_serial);
mk(key_serial, key_essentials, *key);
mk(value_serial, value_essentials, *value);
@@ -118,26 +142,26 @@ void __hot maker::pair(serial_t serial, const buffer &key, buffer &value,
}
}
void maker::setup(const config::actor_params_pod &actor,
void maker::setup(const config::actor_params_pod &actor, unsigned actor_id,
unsigned thread_number) {
key_essentials.flags =
actor.table_flags & (MDBX_INTEGERKEY | MDBX_REVERSEKEY);
assert(actor.keylen_min < UINT8_MAX);
actor.table_flags & (MDBX_INTEGERKEY | MDBX_REVERSEKEY | MDBX_DUPSORT);
assert(actor.keylen_min <= UINT8_MAX);
key_essentials.minlen = (uint8_t)actor.keylen_min;
assert(actor.keylen_max < UINT16_MAX);
assert(actor.keylen_max <= UINT16_MAX);
key_essentials.maxlen = (uint16_t)actor.keylen_max;
value_essentials.flags =
actor.table_flags & (MDBX_INTEGERDUP | MDBX_REVERSEDUP);
assert(actor.datalen_min < UINT8_MAX);
assert(actor.datalen_min <= UINT8_MAX);
value_essentials.minlen = (uint8_t)actor.datalen_min;
assert(actor.datalen_max < UINT16_MAX);
assert(actor.datalen_max <= UINT16_MAX);
value_essentials.maxlen = (uint16_t)actor.datalen_max;
assert(thread_number < 2);
(void)thread_number;
mapping = actor.keygen;
salt = actor.keygen.seed * UINT64_C(14653293970879851569);
salt = (actor.keygen.seed + actor_id) * UINT64_C(14653293970879851569);
// FIXME: TODO
base = 0;
@@ -165,7 +189,7 @@ bool maker::increment(serial_t &serial, int delta) {
//-----------------------------------------------------------------------------
size_t length(serial_t serial) {
static size_t length(serial_t serial) {
size_t n = 0;
if (serial > UINT32_MAX) {
n = 4;
@@ -199,7 +223,10 @@ void __hot maker::mk(const serial_t serial, const essentials &params,
assert(params.maxlen >= length(serial));
out.value.iov_base = out.bytes;
out.value.iov_len = params.minlen;
out.value.iov_len =
(params.maxlen > params.minlen)
? params.minlen + serial % (params.maxlen - params.minlen)
: params.minlen;
if (params.flags & (MDBX_INTEGERKEY | MDBX_INTEGERDUP)) {
assert(params.maxlen == params.minlen);

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -44,7 +44,7 @@ namespace keygen {
* - абсолютное значение ключей или разность между отдельными значениями;
*
* Соответственно, в общих чертах, схема генерации следующая:
* - вводится плоская одномерная "координата" uint64_t;
* - вводится плоская одномерная "координата" serial (uint64_t);
* - генерация специфических паттернов (последовательностей)
* реализуется посредством соответствующих преобразований "координат", при
* этом все подобные преобразования выполняются только над "координатой";
@@ -74,7 +74,7 @@ typedef uint64_t serial_t;
enum : serial_t {
serial_minwith = 8,
serial_maxwith = sizeof(serial_t) * 8,
serial_allones = ~(serial_t)0
serial_allones = ~(serial_t)0u
};
struct result {
@@ -85,6 +85,10 @@ struct result {
uint32_t u32;
uint64_t u64;
};
std::string as_string() const {
return std::string((const char *)value.iov_base, value.iov_len);
}
};
//-----------------------------------------------------------------------------
@@ -115,11 +119,10 @@ public:
void pair(serial_t serial, const buffer &key, buffer &value,
serial_t value_age);
void setup(const config::actor_params_pod &actor, unsigned thread_number);
void setup(const config::actor_params_pod &actor, unsigned actor_id,
unsigned thread_number);
bool increment(serial_t &serial, int delta);
};
size_t length(serial_t serial);
} /* namespace keygen */

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -37,6 +37,31 @@ void __noreturn failure_perror(const char *what, int errnum) {
//-----------------------------------------------------------------------------
static void mdbx_logger(int type, const char *function, int line,
const char *msg, va_list args) {
logging::loglevel level = logging::info;
if (type & MDBX_DBG_EXTRA)
level = logging::extra;
if (type & MDBX_DBG_TRACE)
level = logging::trace;
if (type & MDBX_DBG_PRINT)
level = logging::verbose;
if (!function)
function = "unknown";
if (type & MDBX_DBG_ASSERT) {
log_error("mdbx: assertion failure: %s, %d", function, line);
level = logging::failure;
}
if (logging::output(
level,
strncmp(function, "mdbx_", 5) == 0 ? "%s: " : "mdbx: %s: ", function))
logging::feed_ap(msg, args);
if (type & MDBX_DBG_ASSERT)
abort();
}
namespace logging {
static std::string prefix;
@@ -44,8 +69,19 @@ static std::string suffix;
static loglevel level;
static FILE *last;
void setup(loglevel _level, const std::string &_prefix) {
void setlevel(loglevel _level) {
level = (_level > error) ? failure : _level;
int mdbx_dbg_opts = MDBX_DBG_ASSERT | MDBX_DBG_JITTER | MDBX_DBG_DUMP;
if (level <= trace)
mdbx_dbg_opts |= MDBX_DBG_TRACE;
if (level <= verbose)
mdbx_dbg_opts |= MDBX_DBG_PRINT;
int rc = mdbx_setup_debug(mdbx_dbg_opts, mdbx_logger);
log_trace("set mdbx debug-opts: 0x%02x", rc);
}
void setup(loglevel _level, const std::string &_prefix) {
setlevel(_level);
prefix = _prefix;
}
@@ -157,7 +193,7 @@ bool output(const logging::loglevel priority, const char *format, va_list ap) {
return true;
}
bool feed(const char *format, va_list ap) {
bool feed_ap(const char *format, va_list ap) {
if (!last)
return false;
@@ -176,7 +212,7 @@ bool feed(const char *format, ...) {
va_list ap;
va_start(ap, format);
feed(format, ap);
feed_ap(format, ap);
va_end(ap);
return true;
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -46,11 +46,12 @@ enum loglevel {
const char *level2str(const loglevel level);
void setup(loglevel level, const std::string &prefix);
void setup(const std::string &prefix);
void setlevel(loglevel level);
bool output(const loglevel priority, const char *format, va_list ap);
bool __printf_args(2, 3)
output(const loglevel priority, const char *format, ...);
bool feed(const char *format, va_list ap);
bool feed_ap(const char *format, va_list ap);
bool __printf_args(1, 2) feed(const char *format, ...);
class local_suffix {

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -35,25 +35,31 @@ void actor_params::set_defaults(const std::string &tmpdir) {
mode_flags = MDBX_NOSUBDIR | MDBX_WRITEMAP | MDBX_MAPASYNC | MDBX_NORDAHEAD |
MDBX_NOMEMINIT | MDBX_COALESCE | MDBX_LIFORECLAIM;
table_flags = MDBX_DUPSORT;
size = 1024 * 1024 * 4;
size_lower = -1;
size_now = 1024 * 1024 * ((table_flags & MDBX_DUPSORT) ? 4 : 256);
size_upper = -1;
shrink_threshold = -1;
growth_step = -1;
pagesize = -1;
keygen.seed = 1;
keygen.keycase = kc_random;
keygen.width = 32;
keygen.mesh = 32;
keygen.width = (table_flags & MDBX_DUPSORT) ? 32 : 64;
keygen.mesh = keygen.width;
keygen.split = keygen.width / 2;
keygen.rotate = 0;
keygen.offset = 0;
keygen.rotate = 3;
keygen.offset = 41;
test_duration = 0;
test_nops = 1000;
nrepeat = 1;
nthreads = 1;
keylen_min = 0;
keylen_max = 42;
datalen_min = 0;
datalen_max = 256;
keylen_min = mdbx_keylen_min();
keylen_max = mdbx_keylen_max();
datalen_min = mdbx_datalen_min();
datalen_max = std::min(mdbx_datalen_max(), 256u * 1024 + 42);
batch_read = 4;
batch_write = 4;
@@ -148,10 +154,53 @@ int main(int argc, char *const argv[]) {
config::mode_bits))
continue;
if (config::parse_option(argc, argv, narg, "table", params.table_flags,
config::table_bits))
config::table_bits)) {
if ((params.table_flags & MDBX_DUPFIXED) == 0)
params.table_flags &= ~MDBX_INTEGERDUP;
if ((params.table_flags & MDBX_DUPSORT) == 0)
params.table_flags &=
~(MDBX_DUPFIXED | MDBX_REVERSEDUP | MDBX_INTEGERDUP);
continue;
if (config::parse_option(argc, argv, narg, "size", params.size,
config::binary, 4096 * 4))
}
if (config::parse_option(argc, argv, narg, "pagesize", params.pagesize,
mdbx_limits_pgsize_min(),
mdbx_limits_pgsize_max())) {
const unsigned keylen_max = params.mdbx_keylen_max();
if (params.keylen_min > keylen_max)
params.keylen_min = keylen_max;
if (params.keylen_max > keylen_max)
params.keylen_max = keylen_max;
const unsigned datalen_max = params.mdbx_datalen_max();
if (params.datalen_min > datalen_max)
params.datalen_min = datalen_max;
if (params.datalen_max > datalen_max)
params.datalen_max = datalen_max;
continue;
}
if (config::parse_option(argc, argv, narg, "size-lower", params.size_lower,
mdbx_limits_dbsize_min(params.pagesize),
mdbx_limits_dbsize_max(params.pagesize)))
continue;
if (config::parse_option(argc, argv, narg, "size-upper", params.size_upper,
mdbx_limits_dbsize_min(params.pagesize),
mdbx_limits_dbsize_max(params.pagesize)))
continue;
if (config::parse_option(argc, argv, narg, "size", params.size_now,
mdbx_limits_dbsize_min(params.pagesize),
mdbx_limits_dbsize_max(params.pagesize)))
continue;
if (config::parse_option(
argc, argv, narg, "shrink-threshold", params.shrink_threshold, 0,
(int)std::min((intptr_t)INT_MAX,
mdbx_limits_dbsize_max(params.pagesize) -
mdbx_limits_dbsize_min(params.pagesize))))
continue;
if (config::parse_option(
argc, argv, narg, "growth-step", params.growth_step, 0,
(int)std::min((intptr_t)INT_MAX,
mdbx_limits_dbsize_max(params.pagesize) -
mdbx_limits_dbsize_min(params.pagesize))))
continue;
if (config::parse_option(argc, argv, narg, "keygen.width",
@@ -188,20 +237,39 @@ int main(int argc, char *const argv[]) {
config::duration, 1))
continue;
if (config::parse_option(argc, argv, narg, "keylen.min", params.keylen_min,
config::no_scale, 0, params.keylen_max))
config::no_scale, params.mdbx_keylen_min(),
params.mdbx_keylen_max())) {
if ((params.table_flags & MDBX_INTEGERKEY) ||
params.keylen_max < params.keylen_min)
params.keylen_max = params.keylen_min;
continue;
}
if (config::parse_option(argc, argv, narg, "keylen.max", params.keylen_max,
config::no_scale, params.keylen_min,
mdbx_get_maxkeysize(0)))
config::no_scale, params.mdbx_keylen_min(),
params.mdbx_keylen_max())) {
if ((params.table_flags & MDBX_INTEGERKEY) ||
params.keylen_min > params.keylen_max)
params.keylen_min = params.keylen_max;
continue;
}
if (config::parse_option(argc, argv, narg, "datalen.min",
params.datalen_min, config::no_scale, 0,
params.datalen_max))
params.datalen_min, config::no_scale,
params.mdbx_datalen_min(),
params.mdbx_datalen_max())) {
if ((params.table_flags & MDBX_DUPFIXED) ||
params.datalen_max < params.datalen_min)
params.datalen_max = params.datalen_min;
continue;
}
if (config::parse_option(argc, argv, narg, "datalen.max",
params.datalen_max, config::no_scale,
params.datalen_min, MDBX_MAXDATASIZE))
params.mdbx_datalen_min(),
params.mdbx_datalen_max())) {
if ((params.table_flags & MDBX_DUPFIXED) ||
params.datalen_min > params.datalen_max)
params.datalen_min = params.datalen_max;
continue;
}
if (config::parse_option(argc, argv, narg, "batch.read", params.batch_read,
config::no_scale, 1))
continue;
@@ -269,6 +337,10 @@ int main(int argc, char *const argv[]) {
configure_actor(last_space_id, ac_deadwrite, value, params);
continue;
}
if (config::parse_option(argc, argv, narg, "copy", nullptr)) {
configure_actor(last_space_id, ac_copy, value, params);
continue;
}
if (config::parse_option(argc, argv, narg, "failfast",
global::config::failfast))
continue;

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -182,6 +182,9 @@ void osal_killall_actors(void) {
}
int osal_actor_poll(mdbx_pid_t &pid, unsigned timeout) {
struct timespec ts;
ts.tv_nsec = 0;
ts.tv_sec = (timeout > INT_MAX) ? INT_MAX : timeout;
retry:
int status, options = WNOHANG;
#ifdef WUNTRACED
@@ -209,9 +212,16 @@ retry:
}
if (pid == 0) {
if (timeout && sleep(timeout))
/* child still running */
if (ts.tv_sec == 0 && ts.tv_nsec == 0)
ts.tv_nsec = 1;
if (nanosleep(&ts, &ts) == 0) {
/* timeout and no signal fomr child */
pid = 0;
return 0;
}
if (errno == EINTR)
goto retry;
return 0;
}
switch (errno) {
@@ -291,3 +301,7 @@ std::string osal_tempdir(void) {
return "/dev/shm/";
return "";
}
int osal_removefile(const std::string &pathname) {
return unlink(pathname.c_str()) ? errno : MDBX_SUCCESS;
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -53,7 +53,7 @@ void osal_wait4barrier(void) {
}
}
static HANDLE make_inharitable(HANDLE hHandle) {
static HANDLE make_inheritable(HANDLE hHandle) {
assert(hHandle != NULL && hHandle != INVALID_HANDLE_VALUE);
if (!DuplicateHandle(GetCurrentProcess(), hHandle, GetCurrentProcess(),
&hHandle, 0, TRUE,
@@ -71,7 +71,7 @@ void osal_setup(const std::vector<actor_config> &actors) {
HANDLE hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (!hEvent)
failure_perror("CreateEvent()", GetLastError());
hEvent = make_inharitable(hEvent);
hEvent = make_inheritable(hEvent);
log_trace("osal_setup: event %" PRIuPTR " -> %p", i, hEvent);
events[i] = hEvent;
}
@@ -79,12 +79,12 @@ void osal_setup(const std::vector<actor_config> &actors) {
hBarrierSemaphore = CreateSemaphore(NULL, 0, (LONG)actors.size(), NULL);
if (!hBarrierSemaphore)
failure_perror("CreateSemaphore(BarrierSemaphore)", GetLastError());
hBarrierSemaphore = make_inharitable(hBarrierSemaphore);
hBarrierSemaphore = make_inheritable(hBarrierSemaphore);
hBarrierEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (!hBarrierEvent)
failure_perror("CreateEvent(BarrierEvent)", GetLastError());
hBarrierEvent = make_inharitable(hBarrierEvent);
hBarrierEvent = make_inheritable(hBarrierEvent);
}
void osal_broadcast(unsigned id) {
@@ -262,15 +262,24 @@ int osal_actor_start(const actor_config &config, mdbx_pid_t &pid) {
STARTUPINFOA StartupInfo;
GetStartupInfoA(&StartupInfo);
char exename[_MAX_PATH];
char exename[_MAX_PATH + 1];
DWORD exename_size = sizeof(exename);
if (!QueryFullProcessImageNameA(GetCurrentProcess(), 0, exename,
&exename_size))
failure_perror("QueryFullProcessImageName()", GetLastError());
std::string cmdline = "test_mdbx.child ";
if (exename[1] != ':') {
exename_size = GetModuleFileName(NULL, exename, sizeof(exename));
if (exename_size >= sizeof(exename))
return ERROR_BAD_LENGTH;
}
std::string cmdline = "$ ";
ArgvQuote(cmdline, thunk_param(config));
if (cmdline.size() >= 32767)
return ERROR_BAD_LENGTH;
PROCESS_INFORMATION ProcessInformation;
if (!CreateProcessA(exename, const_cast<char *>(cmdline.c_str()),
NULL, // Retuned process handle is not inheritable.
@@ -280,7 +289,7 @@ int osal_actor_start(const actor_config &config, mdbx_pid_t &pid) {
NULL, // Inherit the parent's environment.
NULL, // Inherit the parent's current directory.
&StartupInfo, &ProcessInformation))
return GetLastError();
failure_perror(exename, GetLastError());
CloseHandle(ProcessInformation.hThread);
pid = ProcessInformation.dwProcessId;
@@ -396,3 +405,7 @@ std::string osal_tempdir(void) {
DWORD len = GetTempPathA(sizeof(buf), buf);
return std::string(buf, len);
}
int osal_removefile(const std::string &pathname) {
return DeleteFileA(pathname.c_str()) ? MDBX_SUCCESS : GetLastError();
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -32,6 +32,7 @@ void osal_udelay(unsigned us);
void osal_yield(void);
bool osal_istty(int fd);
std::string osal_tempdir(void);
int osal_removefile(const std::string &pathname);
#ifdef _MSC_VER
#ifndef STDIN_FILENO

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -31,6 +31,8 @@ const char *testcase2str(const actor_testcase testcase) {
return "jitter";
case ac_try:
return "try";
case ac_copy:
return "copy";
}
}
@@ -68,31 +70,6 @@ const char *keygencase2str(const keygen_case keycase) {
//-----------------------------------------------------------------------------
static void mdbx_logger(int type, const char *function, int line,
const char *msg, va_list args) {
logging::loglevel level = logging::info;
if (type & MDBX_DBG_EXTRA)
level = logging::extra;
if (type & MDBX_DBG_TRACE)
level = logging::trace;
if (type & MDBX_DBG_PRINT)
level = logging::verbose;
if (!function)
function = "unknown";
if (type & MDBX_DBG_ASSERT) {
log_error("mdbx: assertion failure: %s, %d", function, line);
level = logging::failure;
}
if (logging::output(
level,
strncmp(function, "mdbx_", 5) == 0 ? "%s: " : "mdbx: %s: ", function))
logging::feed(msg, args);
if (type & MDBX_DBG_ASSERT)
abort();
}
int testcase::oom_callback(MDBX_env *env, int pid, mdbx_tid_t tid, uint64_t txn,
unsigned gap, int retry) {
@@ -117,16 +94,8 @@ void testcase::db_prepare() {
log_trace(">> db_prepare");
assert(!db_guard);
int mdbx_dbg_opts = MDBX_DBG_ASSERT | MDBX_DBG_JITTER | MDBX_DBG_DUMP;
if (config.params.loglevel <= logging::trace)
mdbx_dbg_opts |= MDBX_DBG_TRACE;
if (config.params.loglevel <= logging::verbose)
mdbx_dbg_opts |= MDBX_DBG_PRINT;
int rc = mdbx_setup_debug(mdbx_dbg_opts, mdbx_logger);
log_trace("set mdbx debug-opts: 0x%02x", rc);
MDBX_env *env = nullptr;
rc = mdbx_env_create(&env);
int rc = mdbx_env_create(&env);
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_env_create()", rc);
@@ -149,7 +118,10 @@ void testcase::db_prepare() {
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_env_set_oomfunc()", rc);
rc = mdbx_env_set_mapsize(env, (size_t)config.params.size);
rc = mdbx_env_set_geometry(
env, config.params.size_lower, config.params.size_now,
config.params.size_upper, config.params.growth_step,
config.params.shrink_threshold, config.params.pagesize);
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_env_set_mapsize()", rc);
@@ -473,6 +445,9 @@ bool test_execute(const actor_config &config) {
case ac_try:
test.reset(new testcase_try(config, pid));
break;
case ac_copy:
test.reset(new testcase_copy(config, pid));
break;
default:
test.reset(new testcase(config, pid));
break;

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -203,3 +203,14 @@ public:
bool run();
bool teardown();
};
class testcase_copy : public testcase {
const std::string copy_pathname;
void copy_db(const bool with_compaction);
public:
testcase_copy(const actor_config &config, const mdbx_pid_t pid)
: testcase(config, pid),
copy_pathname(config.params.pathname_db + "-copy") {}
bool run();
};

View File

@@ -78,21 +78,25 @@
<LinkIncremental>true</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<LinkIncremental>true</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<LinkIncremental>false</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>
@@ -180,6 +184,7 @@
<ClCompile Include="cases.cc" />
<ClCompile Include="chrono.cc" />
<ClCompile Include="config.cc" />
<ClCompile Include="copy.cc" />
<ClCompile Include="dead.cc" />
<ClCompile Include="hill.cc" />
<ClCompile Include="try.cc" />

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*
@@ -93,7 +93,7 @@ bool hex2data(const char *hex_begin, const char *hex_end, void *ptr,
//-----------------------------------------------------------------------------
/* TODO: replace my 'libmera' fomr t1ha. */
/* TODO: replace my 'libmera' from t1ha. */
uint64_t entropy_ticks(void) {
#if defined(EMSCRIPTEN)
return (uint64_t)emscripten_get_now();

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
*

View File

@@ -4,7 +4,7 @@
*/
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2012-2015 Howard Chu, Symas Corp.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.

View File

@@ -5,7 +5,7 @@
/*
* Copyright 2017 Ilya Shipitsin <chipitsine@gmail.com>.
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2012-2015 Howard Chu, Symas Corp.
* All rights reserved.
*