Compare commits

...

56 Commits

Author SHA1 Message Date
moneromooo-monero
a3aa2b5a57 mdbx-doc: import - mdb_cursor_del does not invalidate the cursor (ITS#8857). 2018-09-24 19:35:39 +03:00
Howard Chu
02276500c9 mdbx-doc: import - GET_MULTIPLE etc don't return the key (ITS#8908).
Unnecessary since these are DUPs, the key will always be the same
2018-09-24 19:35:36 +03:00
Leonid Yuriev
de44ecccd1 mdbx: backport - update MAX_PAGENO and MAX_MAPSIZE64. 2018-09-23 18:07:29 +03:00
Leonid Yuriev
5049c86517 mdbx: backport - avoid empty and unneeded large/overflow pages (squashed). 2018-09-23 18:07:29 +03:00
Leonid Yuriev
d2854e0760 mdbx: backport - refine mdbx_chk (squashed).
- refine 'mismatch idl length' error message.
 - add/fix printf-format checking.
 - refine dbi-structure.
2018-09-23 18:07:29 +03:00
Leonid Yuriev
5a29214ad9 mdbx-test: backport - update 'gc.sh' script (squashed). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
b51d92d449 mdbx-test: backport - update test (squashed).
- add support for 'default' options values.
 - add min/max cases for option values.
 - add support for db-geometry params.
 - fix int-types for 32-bit builds (minor).
 - fix key/value generation for long-length cases.
 - fix update_flags for non-MDBX_DUPSORT.
 - 'none' for config-verbs.
 - check commandline length under Windows.
 - workaround for QueryFullProcessImageNameA() bug.
 - add setloglevel().
 - workaroung for MSVC bug.
 - avoid extra 'jitter' testcase loops.
 - cleanup DUPSORT flags.
 - refine key/value min/max handling.
 - dump keygen params.
 - fix/refine keygen.
 - alter keygen defaults (rotate 3, offset 41).
 - default test-db size 4mb or 256mb.
 - fix/refine keygen for non-MDBX_DUPSORT.
 - seeding keygen with actor_id for better spreading.
2018-09-23 15:39:56 +03:00
Leonid Yuriev
6da477d37f mdbx-ci: backport - refines for Windows (squashed).
- push logs to appveyor separately.
 - rename 'test.exe' to 'mdbx_test.exe'.
 - add test.db to appveyor artefacts (windows).
2018-09-23 15:39:56 +03:00
Leonid Yuriev
6150a8c903 mdbx: backport - fix/refine mdbx_update_gc() (squashed). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
f3e9731da4 mdbx: backport - move macros/inlines to fix Windows builds.
Change-Id: I48aaf6b77466bb8b13294b84de73fb6063c88190
2018-09-23 15:39:56 +03:00
Leonid Yuriev
353b6b8af0 mdbx: backport - refine assections (minor).
Change-Id: Ic924988b8ce043d6106df381c996dd2c8ff9ca1f
2018-09-23 15:39:56 +03:00
Leonid Yuriev
3f10e58df2 mdbx: backport - re-define assert macro via mdbx_assert.
Change-Id: I317801ba4200bdf1aa5cacf75d21a8e633fbc48a
2018-09-23 15:39:56 +03:00
Leonid Yuriev
d232737087 mdbx: backport - add MDBX_FORCE_ASSERT.
Change-Id: I68a9f7b42663ea157c7c0a5a58797c94127b45ed
2018-09-23 15:39:56 +03:00
Leo Yuriev
e32ca55258 mdbx: backport - fix tracking around mdbx_cursor_del(). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
f57ffc987c mdbx: backport - drop inherited broken audit (will be fixed in the master branch).
Internal self-audit (inherited from LMDB) is invalid and useless
for sub-db and dupsort cases.
2018-09-23 15:39:56 +03:00
Leonid Yuriev
cdd510d20e mdbx: backport - prevent DB corruption due rebalance bugs.
Won't fix https://github.com/leo-yuriev/libmdbx/issues/38 in the 'stable/0.1' branch,
but add checks to prevent DB corruption.
2018-09-23 15:39:56 +03:00
Leo Yuriev
d757ba1266 mdbx: backport - fix MDBX_CORRUPTED due open/shrink collision. 2018-09-23 15:39:56 +03:00
Leonid Yuriev
337f7589f8 mdbx: backport - fix mdbx_pnl_search(). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
912728a322 mdbx: backport - fix mdbx_replace().
Change-Id: I2af00f101017795ca2b967479f86e5ea7e8ad37b
2018-09-23 15:39:56 +03:00
Leonid Yuriev
204b5a532d mdbx: backport - shorten maxkeysize (will be fixed in the master branch).
Change-Id: I660b1b3e454d9b51a24d3b4cc987c8e2980bd435
2018-09-23 15:39:56 +03:00
Leonid Yuriev
014be165c3 mdbx: backport - allow GC's PNL be partially unused. 2018-09-23 15:39:56 +03:00
Leonid Yuriev
a9244f807b mdbx: backport - setup mdbx_cmp_memn() as data-comparator for safety. 2018-09-23 15:39:56 +03:00
Leonid Yuriev
6d438605dd mdbx: backport - check comparator for MDBX_GET_BOTH and MDBX_GET_BOTH_RANGE. 2018-09-23 15:39:56 +03:00
Leo Yuriev
34300150a1 mdbx: backport - don't touch mm_psize and mm_flags while provoking bad readers (debug-only). 2018-09-23 15:39:56 +03:00
Leo Yuriev
1b2b98234f mdbx: backport - fix concurrent opening with custom pagesize (get pagesize from meta-page early). 2018-09-23 15:39:56 +03:00
Leonid Yuriev
bd672a5583 mdbx: backport - add mdbx_limits_xyz() (squashed).
Change-Id: I56c79704c59386a0c4d84b001020484c23925e6c
2018-09-23 15:39:56 +03:00
Leonid Yuriev
888003c072 mdbx: backport - fix comments typos (squashed). 2018-09-23 14:40:31 +03:00
Leo Yuriev
bc6db4e4d7 mdbx: backport - allow mdbx_env_compact() to fix page leaks.
Don't treat fixing page leaks as an error while copy DB with compactification.

Change-Id: I2a575ff9e2b24610172aaca939b5f6957c26ec77
2018-09-23 14:40:31 +03:00
Leonid Yuriev
57655583e5 mdbx: backport - fix/rewrite mdbx_update_gc().
Change-Id: I580a1ff0cbeeb529e2bcbd50d97bfba7bcf5a546
2018-08-13 23:30:39 +03:00
Leonid Yuriev
b91e645919 mdbx-test: add 'gc.sh' script.
Change-Id: I633c93c0865b0d2609688713e986edf51ce6547d
2018-08-13 21:44:43 +03:00
Leonid Yuriev
652bb08f8c mdbx-test: backport - use strtoull() and retry with base=10.
Change-Id: Ica846ed0a13eb4468a45620518b9ccf85e77a764
2018-08-13 21:44:10 +03:00
Leonid Yuriev
59026d5f84 mdbx-test: backport - fix minor typos.
Change-Id: I4889a0e698bdfdda7eed257a5cd29e8b8089d102
2018-08-13 21:43:07 +03:00
Leonid Yuriev
e18551061e mdbx-test: backport - fix keylen/datalen min/max ranges checking.
Change-Id: Iee5d2f71ad22ec6e86167f5181deff54f0b5b518
2018-08-13 21:42:44 +03:00
Leonid Yuriev
e054ad2ebb mdbx-test: backport - add 'strikethrough' for bitmask-options.
Change-Id: I86dd2f8cdbd5a32a0471a5eee1e2b3a5857541ac
2018-08-13 21:42:22 +03:00
Leonid Yuriev
bff6aa460a mdbx: backport - fix MDBX_EKEYMISMATCH while update multi-value with MDBX_CURRENT.
Change-Id: I3095620a94f694fb2c29b9c4faab9ea02b9bd7b7
2018-08-13 21:41:17 +03:00
Leonid Yuriev
3979ba4784 mdbx: backport - fix assertions.
Change-Id: I95c43ef1ea2da55a124dc43f03890cf1d96f2e61
2018-08-13 21:40:48 +03:00
Leonid Yuriev
d2fcbf5f82 mdbx: backport - fix assert-condition inside mdbx_pnl_xappend().
Change-Id: Id5ac89c85b7e673c44d60a626c805fe666d221bc
2018-08-10 09:36:44 +03:00
Leo Yuriev
38067c4566 mdbx: backport - fix 'db_dummy' inside mdbx_dbi_open_ex().
Change-Id: I70a21c9b77a43c5af749da5723fa965487a056b0
2018-08-09 13:07:02 +03:00
Leo Yuriev
d4bfc17818 mdbx: backport - add fallback2shared for mdbx_lck_exclusive(). 2018-08-09 13:07:02 +03:00
Leo Yuriev
eb99480253 mdbx: backport - drop unused mdbx_lck_upgrade(). 2018-08-09 13:07:02 +03:00
Leo Yuriev
c9790b28d0 mdbx-cmake: fix so-version.
Change-Id: I427d2f27f9092d65a0ffd11353ca466070e98618
2018-08-09 13:07:02 +03:00
Leo Yuriev
9a1ef8acfb mdbx-cmake: remove warning-message.
Change-Id: Icf9e4f7a96916cf9ab04613344867217be04827a
2018-08-09 13:07:02 +03:00
Leo Yuriev
e442395cbd mdbx: bump version to v0.1.6
Change-Id: I95d45a815008e2cc9a8785a8c762310a1e907e21
2018-07-31 11:43:41 +03:00
Leo Yuriev
e57e521609 mdbx: backport - fix nasty suspend_and_append() bug.
Change-Id: I043adcff2e6c040426a51b5d4b15bac849e6dd9f
2018-07-31 11:43:35 +03:00
Leo Yuriev
d1809e6e2d mdbx: backport - minor fix to avoid Valgrind false-positive issue.
Change-Id: Ifa4dc51b500ff42a88182d750e22572aa5b2155b
2018-07-01 17:43:04 +03:00
Leo Yuriev
c579b974a2 mdbx: backport - avoid weak meta inside mdbx_init_metas().
Change-Id: Ib9c5ab04ad8cff3ad43d94a288cecec45d7ef37d
2018-06-30 02:43:10 +03:00
Leo Yuriev
de43ab0d21 mdbx-ci: migrate to Circle-CI 2.0
Change-Id: Id86af9e033d64a4dc2043db33cd8e7ae173feb22
2018-06-21 19:45:42 +03:00
Leo Yuriev
db50fb8726 mdbx: backport - fix Coverity warning (minor, paranoia).
Change-Id: I232377a03244dc33beb4f332c0024b454027f659
2018-06-21 18:26:31 +03:00
Leo Yuriev
408848a425 mdbx: Merge branch 'master' into stable/0.1
Change-Id: Ide2fbcd1b0b6bacbc4f07049633df81fede397eb
2018-06-21 17:57:06 +03:00
Leo Yuriev
e880e734ce mdbx: Merge branch 'master' into stable/0.1 2018-06-19 14:46:39 +03:00
Leo Yuriev
61b2a7fc54 mdbx: Merge branch 'master' into stable/0.1
Change-Id: I80b5afe1d227009a42096b5c6f2bfa9e5eb09036
2018-06-15 03:46:17 +03:00
Leo Yuriev
e1e17fd6a4 mdbx: Merge branch 'master' into stable/0.1 2018-06-14 13:54:01 +03:00
Leo Yuriev
17d3e7190c mdbx: Merge branch 'master' into stable/0.1
Change-Id: I7460e6a8b42c9bbeadfdf20d326c31c1d4a98969
2018-06-01 16:42:48 +03:00
Leo Yuriev
22d8b0b2e1 mdbx: Merge branch 'master' into stable/0.1
Change-Id: Idd1af2c07b48bfb1f335aa31a52bed4bd68514ac
2018-05-29 03:14:07 +03:00
Leo Yuriev
9384d0efa6 mdbx: Merge branch 'master' into stable/0.1
Change-Id: I3318b330e4aa5fd0db4642af6ef81e69814f0cc5
2018-05-22 12:03:57 +03:00
Leo Yuriev
ffafd5be10 mdbx: disable warning #5045 for MSVC (minor). 2018-05-21 16:36:41 +03:00
27 changed files with 1381 additions and 914 deletions

20
.circleci/config.yml Normal file
View File

@@ -0,0 +1,20 @@
version: 2
jobs:
build:
docker:
- image: circleci/buildpack-deps:artful
environment:
- TESTDB: /tmp/test.db
- TESTLOG: /tmp/test.log
steps:
- checkout
- run: make all
- run: ulimit -c unlimited && make check
- run:
command: |
mkdir -p /tmp/artifacts
mv -t /tmp/artifacts $TESTLOG $TESTDB core.*
when: on_fail
- store_artifacts:
path: /tmp/artifacts
destination: test-artifacts

View File

@@ -2,19 +2,10 @@ cmake_minimum_required(VERSION 2.8.7)
set(TARGET mdbx)
project(${TARGET})
message(WARNING "
***************************************************************
MDBX is under active development, database format and API
aren't stable at least until 2018Q3. New version won't be
backwards compatible. Main focus of the rework is to provide
clear and robust API and new features.
***************************************************************
")
set(MDBX_VERSION_MAJOR 0)
set(MDBX_VERSION_MINOR 1)
set(MDBX_VERSION_RELEASE 3)
set(MDBX_VERSION_REVISION 1)
set(MDBX_VERSION_RELEASE 6)
set(MDBX_VERSION_REVISION 0)
set(MDBX_VERSION_STRING ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}.${MDBX_VERSION_RELEASE})
@@ -111,14 +102,14 @@ add_library(${TARGET}_SHARED SHARED
set_target_properties(${TARGET}_SHARED PROPERTIES
VERSION ${MDBX_VERSION_STRING}
SOVERSION ${MDBX_VERSION_MAJOR}
SOVERSION ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}
OUTPUT_NAME ${TARGET}
CLEAN_DIRECT_OUTPUT 1
)
set_target_properties(${TARGET}_STATIC PROPERTIES
VERSION ${MDBX_VERSION_STRING}
SOVERSION ${MDBX_VERSION_MAJOR}
SOVERSION ${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}
OUTPUT_NAME ${TARGET}
CLEAN_DIRECT_OUTPUT 1
)

View File

@@ -1,4 +1,4 @@
version: 0.1.5.{build}
version: 0.1.6.{build}
environment:
matrix:
@@ -32,21 +32,23 @@ build_script:
test_script:
- ps: |
if (($env:PLATFORM -eq "x86") -and (Test-Path "C:\projects\libmdbx\Win32\$env:CONFIGURATION\test.exe" -PathType Leaf)) {
$test = "C:\projects\libmdbx\Win32\$env:CONFIGURATION\test.exe"
if (($env:PLATFORM -eq "x86") -and (Test-Path "C:\projects\libmdbx\Win32\$env:CONFIGURATION\mdbx_test.exe" -PathType Leaf)) {
$mdbx_test = "C:\projects\libmdbx\Win32\$env:CONFIGURATION\mdbx_test.exe"
$mdbx_chk = "C:\projects\libmdbx\Win32\$env:CONFIGURATION\mdbx_chk.exe"
} elseif (($env:PLATFORM -ne "ARM") -and ($env:PLATFORM -ne "ARM64")) {
$test = "C:\projects\libmdbx\$env:PLATFORM\$env:CONFIGURATION\test.exe"
$mdbx_test = "C:\projects\libmdbx\$env:PLATFORM\$env:CONFIGURATION\mdbx_test.exe"
$mdbx_chk = "C:\projects\libmdbx\$env:PLATFORM\$env:CONFIGURATION\mdbx_chk.exe"
} else {
$test = ""
$mdbx_test = ""
$mdbx_chk = ""
}
if ($test -ne "") {
& "$test" --pathname=tmp.db --dont-cleanup-after basic | Tee-Object -file test.log | Select-Object -last 42
& "$mdbx_chk" -nvv tmp.db | Tee-Object -file chk.log | Select-Object -last 42
if ($mdbx_test -ne "") {
& "$mdbx_test" --pathname=test.db --dont-cleanup-after basic | Tee-Object -file test.log | Select-Object -last 42
& "$mdbx_chk" -nvv test.db | Tee-Object -file chk.log | Select-Object -last 42
}
on_failure:
- ps: Push-AppveyorArtifact test.log chk.log
- ps: Push-AppveyorArtifact test.log
- ps: Push-AppveyorArtifact test.db
- ps: Push-AppveyorArtifact chk.log

View File

@@ -1,14 +0,0 @@
machine:
timezone:
Europe/Moscow
database:
override:
compile:
override:
- make all
test:
override:
- make check || mv test.log ${CIRCLE_ARTIFACTS}/

26
mdbx.h
View File

@@ -344,17 +344,17 @@ typedef enum MDBX_cursor_op {
MDBX_GET_BOTH, /* MDBX_DUPSORT-only: Position at key/data pair. */
MDBX_GET_BOTH_RANGE, /* MDBX_DUPSORT-only: position at key, nearest data. */
MDBX_GET_CURRENT, /* Return key/data at current cursor position */
MDBX_GET_MULTIPLE, /* MDBX_DUPFIXED-only: Return key and up to a page of
* duplicate data items from current cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE.*/
MDBX_GET_MULTIPLE, /* MDBX_DUPFIXED-only: Return up to a page of duplicate
* data items from current cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE. */
MDBX_LAST, /* Position at last key/data item */
MDBX_LAST_DUP, /* MDBX_DUPSORT-only: Position at last data item
* of current key. */
MDBX_NEXT, /* Position at next data item */
MDBX_NEXT_DUP, /* MDBX_DUPSORT-only: Position at next data item
* of current key. */
MDBX_NEXT_MULTIPLE, /* MDBX_DUPFIXED-only: Return key and up to a page of
* duplicate data items from next cursor position.
MDBX_NEXT_MULTIPLE, /* MDBX_DUPFIXED-only: Return up to a page of duplicate
* data items from next cursor position.
* Move cursor to prepare for MDBX_NEXT_MULTIPLE. */
MDBX_NEXT_NODUP, /* Position at first data item of next key */
MDBX_PREV, /* Position at previous data item */
@@ -366,7 +366,7 @@ typedef enum MDBX_cursor_op {
MDBX_SET_RANGE, /* Position at first key greater than or equal to
* specified key. */
MDBX_PREV_MULTIPLE /* MDBX_DUPFIXED-only: Position at previous page and
* return key and up to a page of duplicate data items. */
* return up to a page of duplicate data items. */
} MDBX_cursor_op;
/* Return Codes
@@ -468,7 +468,7 @@ typedef struct MDBX_envinfo {
uint64_t lower; /* lower limit for datafile size */
uint64_t upper; /* upper limit for datafile size */
uint64_t current; /* current datafile size */
uint64_t shrink; /* shrink theshold for datafile */
uint64_t shrink; /* shrink threshold for datafile */
uint64_t grow; /* growth step for datafile */
} mi_geo;
uint64_t mi_mapsize; /* Size of the data memory map */
@@ -900,7 +900,7 @@ LIBMDBX_API int mdbx_env_set_maxdbs(MDBX_env *env, MDBX_dbi dbs);
*
* Returns The maximum size of a key we can write. */
LIBMDBX_API int mdbx_env_get_maxkeysize(MDBX_env *env);
LIBMDBX_API int mdbx_get_maxkeysize(size_t pagesize);
LIBMDBX_API int mdbx_get_maxkeysize(intptr_t pagesize);
/* Set application information associated with the MDBX_env.
*
@@ -966,7 +966,7 @@ LIBMDBX_API int mdbx_env_set_assert(MDBX_env *env, MDBX_assert_func *func);
* Returns A non-zero error value on failure and 0 on success, some
* possible errors are:
* - MDBX_PANIC - a fatal error occurred earlier and the environment
* must be shut down.
* must be shut down.
* - MDBX_MAP_RESIZED - another process wrote data beyond this MDBX_env's
* mapsize and this environment's map must be resized
* as well. See mdbx_env_set_mapsize().
@@ -1461,6 +1461,9 @@ LIBMDBX_API int mdbx_cursor_put(MDBX_cursor *cursor, MDBX_val *key,
/* Delete current key/data pair
*
* This function deletes the key/data pair to which the cursor refers.
* This does not invalidate the cursor, so operations such as MDBX_NEXT
* can still be used on it. Both MDBX_NEXT and MDBX_GET_CURRENT will return
* the same record after this operation.
*
* [in] cursor A cursor handle returned by mdbx_cursor_open()
* [in] flags Options for this operation. This parameter must be set to 0
@@ -1673,6 +1676,11 @@ LIBMDBX_API int mdbx_is_dirty(const MDBX_txn *txn, const void *ptr);
LIBMDBX_API int mdbx_dbi_sequence(MDBX_txn *txn, MDBX_dbi dbi, uint64_t *result,
uint64_t increment);
LIBMDBX_API int mdbx_limits_pgsize_min(void);
LIBMDBX_API int mdbx_limits_pgsize_max(void);
LIBMDBX_API intptr_t mdbx_limits_dbsize_min(intptr_t pagesize);
LIBMDBX_API intptr_t mdbx_limits_dbsize_max(intptr_t pagesize);
/*----------------------------------------------------------------------------*/
/* attribute support functions for Nexenta */
typedef uint_fast64_t mdbx_attr_t;

View File

@@ -160,7 +160,7 @@
* size up to 2^44 bytes, in case of 4K pages. */
typedef uint32_t pgno_t;
#define PRIaPGNO PRIu32
#define MAX_PAGENO ((pgno_t)UINT64_C(0xffffFFFFffff))
#define MAX_PAGENO UINT32_C(0x7FFFffff)
#define MIN_PAGENO NUM_METAS
/* A transaction ID. */
@@ -389,9 +389,7 @@ typedef struct MDBX_page {
#else
#define MAX_MAPSIZE32 UINT32_C(0x7ff80000)
#endif
#define MAX_MAPSIZE64 \
((sizeof(pgno_t) > 4) ? UINT64_C(0x7fffFFFFfff80000) \
: MAX_PAGENO * (uint64_t)MAX_PAGESIZE)
#define MAX_MAPSIZE64 (MAX_PAGENO * (uint64_t)MAX_PAGESIZE)
#define MAX_MAPSIZE ((sizeof(size_t) < 8) ? MAX_MAPSIZE32 : MAX_MAPSIZE64)
@@ -408,7 +406,7 @@ typedef struct MDBX_lockinfo {
volatile uint32_t mti_envmode;
#ifdef MDBX_OSAL_LOCK
/* Mutex protecting write access to this table. */
/* Mutex protecting write-txn. */
union {
MDBX_OSAL_LOCK mti_wmutex;
uint8_t pad_mti_wmutex[MDBX_OSAL_LOCK_SIZE % sizeof(size_t)];
@@ -751,8 +749,8 @@ struct MDBX_env {
MDBX_PNL me_free_pgs;
/* ID2L of pages written during a write txn. Length MDBX_PNL_UM_SIZE. */
MDBX_ID2L me_dirtylist;
/* Max number of freelist items that can fit in a single overflow page */
unsigned me_maxfree_1pg;
/* Number of freelist items that can fit in a single overflow page */
unsigned me_maxgc_ov1page;
/* Max size of a node on a page */
unsigned me_nodemax;
unsigned me_maxkey_limit; /* max size of a key */
@@ -817,15 +815,12 @@ void mdbx_panic(const char *fmt, ...)
#define mdbx_assert_enabled() unlikely(mdbx_runtime_flags &MDBX_DBG_ASSERT)
#define mdbx_audit_enabled() unlikely(mdbx_runtime_flags &MDBX_DBG_AUDIT)
#define mdbx_debug_enabled(type) \
unlikely(mdbx_runtime_flags &(type & (MDBX_DBG_TRACE | MDBX_DBG_EXTRA)))
#else
#define mdbx_debug_enabled(type) (0)
#define mdbx_audit_enabled() (0)
#ifndef NDEBUG
#if !defined(NDEBUG) || defined(MDBX_FORCE_ASSERT)
#define mdbx_assert_enabled() (1)
#else
#define mdbx_assert_enabled() (0)
@@ -931,17 +926,8 @@ void mdbx_panic(const char *fmt, ...)
/* assert(3) variant in transaction context */
#define mdbx_tassert(txn, expr) mdbx_assert((txn)->mt_env, expr)
static __inline void mdbx_jitter4testing(bool tiny) {
#ifndef NDEBUG
if (MDBX_DBG_JITTER & mdbx_runtime_flags)
mdbx_osal_jitter(tiny);
#else
(void)tiny;
#endif
}
/*----------------------------------------------------------------------------*/
/* Internal prototypes and inlines */
/* Internal prototypes */
int mdbx_reader_check0(MDBX_env *env, int rlocked, int *dead);
int mdbx_rthc_alloc(mdbx_thread_key_t *key, MDBX_reader *begin,
@@ -952,24 +938,6 @@ void mdbx_rthc_global_init(void);
void mdbx_rthc_global_dtor(void);
void mdbx_rthc_thread_dtor(void *ptr);
static __inline bool mdbx_is_power2(size_t x) { return (x & (x - 1)) == 0; }
static __inline size_t mdbx_roundup2(size_t value, size_t granularity) {
assert(mdbx_is_power2(granularity));
return (value + granularity - 1) & ~(granularity - 1);
}
static __inline unsigned mdbx_log2(size_t value) {
assert(mdbx_is_power2(value));
unsigned log = 0;
while (value > 1) {
log += 1;
value >>= 1;
}
return log;
}
#define MDBX_IS_ERROR(rc) \
((rc) != MDBX_RESULT_TRUE && (rc) != MDBX_RESULT_FALSE)
@@ -1043,6 +1011,8 @@ static __inline unsigned mdbx_log2(size_t value) {
/* Test if a page is a sub page */
#define IS_SUBP(p) F_ISSET((p)->mp_flags, P_SUBP)
#define PAGETYPE(p) ((p)->mp_flags & (P_BRANCH | P_LEAF | P_LEAF2 | P_OVERFLOW))
/* The number of overflow pages needed to store the given size. */
#define OVPAGES(env, size) (bytes2pgno(env, PAGEHDRSZ - 1 + (size)) + 1)
@@ -1113,71 +1083,12 @@ typedef struct MDBX_node {
* This is node header plus key plus data size. */
#define LEAFSIZE(k, d) (NODESIZE + (k)->iov_len + (d)->iov_len)
/* Address of node i in page p */
static __inline MDBX_node *NODEPTR(MDBX_page *p, unsigned i) {
assert(NUMKEYS(p) > (unsigned)(i));
return (MDBX_node *)((char *)(p) + (p)->mp_ptrs[i] + PAGEHDRSZ);
}
/* Address of the key for the node */
#define NODEKEY(node) (void *)((node)->mn_data)
/* Address of the data for a node */
#define NODEDATA(node) (void *)((char *)(node)->mn_data + (node)->mn_ksize)
/* Get the page number pointed to by a branch node */
static __inline pgno_t NODEPGNO(const MDBX_node *node) {
pgno_t pgno;
if (UNALIGNED_OK) {
pgno = node->mn_ksize_and_pgno;
if (sizeof(pgno_t) > 4)
pgno &= MAX_PAGENO;
} else {
pgno = node->mn_lo | ((pgno_t)node->mn_hi << 16);
if (sizeof(pgno_t) > 4)
pgno |= ((uint64_t)node->mn_flags) << 32;
}
return pgno;
}
/* Set the page number in a branch node */
static __inline void SETPGNO(MDBX_node *node, pgno_t pgno) {
assert(pgno <= MAX_PAGENO);
if (UNALIGNED_OK) {
if (sizeof(pgno_t) > 4)
pgno |= ((uint64_t)node->mn_ksize) << 48;
node->mn_ksize_and_pgno = pgno;
} else {
node->mn_lo = (uint16_t)pgno;
node->mn_hi = (uint16_t)(pgno >> 16);
if (sizeof(pgno_t) > 4)
node->mn_flags = (uint16_t)((uint64_t)pgno >> 32);
}
}
/* Get the size of the data in a leaf node */
static __inline size_t NODEDSZ(const MDBX_node *node) {
size_t size;
if (UNALIGNED_OK) {
size = node->mn_dsize;
} else {
size = node->mn_lo | ((size_t)node->mn_hi << 16);
}
return size;
}
/* Set the size of the data for a leaf node */
static __inline void SETDSZ(MDBX_node *node, size_t size) {
assert(size < INT_MAX);
if (UNALIGNED_OK) {
node->mn_dsize = (uint32_t)size;
} else {
node->mn_lo = (uint16_t)size;
node->mn_hi = (uint16_t)(size >> 16);
}
}
/* The size of a key in a node */
#define NODEKSZ(node) ((node)->mn_ksize)
@@ -1230,19 +1141,8 @@ static __inline void SETDSZ(MDBX_node *node, size_t size) {
#define mdbx_cmp2int(a, b) (((a) > (b)) - ((b) > (a)))
#endif
static __inline size_t pgno2bytes(const MDBX_env *env, pgno_t pgno) {
mdbx_assert(env, (1u << env->me_psize2log) == env->me_psize);
return ((size_t)pgno) << env->me_psize2log;
}
static __inline MDBX_page *pgno2page(const MDBX_env *env, pgno_t pgno) {
return (MDBX_page *)(env->me_map + pgno2bytes(env, pgno));
}
static __inline pgno_t bytes2pgno(const MDBX_env *env, size_t bytes) {
mdbx_assert(env, (env->me_psize >> env->me_psize2log) == 1);
return (pgno_t)(bytes >> env->me_psize2log);
}
/* Do not spill pages to disk if txn is getting full, may fail instead */
#define MDBX_NOSPILL 0x8000
static __inline pgno_t pgno_add(pgno_t base, pgno_t augend) {
assert(base <= MAX_PAGENO);
@@ -1254,10 +1154,11 @@ static __inline pgno_t pgno_sub(pgno_t base, pgno_t subtrahend) {
return (subtrahend < base - MIN_PAGENO) ? base - subtrahend : MIN_PAGENO;
}
static __inline size_t pgno_align2os_bytes(const MDBX_env *env, pgno_t pgno) {
return mdbx_roundup2(pgno2bytes(env, pgno), env->me_os_psize);
}
static __inline pgno_t pgno_align2os_pgno(const MDBX_env *env, pgno_t pgno) {
return bytes2pgno(env, pgno_align2os_bytes(env, pgno));
static __inline void mdbx_jitter4testing(bool tiny) {
#ifndef NDEBUG
if (MDBX_DBG_JITTER & mdbx_runtime_flags)
mdbx_osal_jitter(tiny);
#else
(void)tiny;
#endif
}

View File

@@ -68,11 +68,19 @@ static int mdbx_lck_op(mdbx_filehandle_t fd, int op, int lck, off_t offset,
}
}
static __inline int mdbx_lck_exclusive(int lfd) {
static __inline int mdbx_lck_exclusive(int lfd, bool fallback2shared) {
assert(lfd != INVALID_HANDLE_VALUE);
if (flock(lfd, LOCK_EX | LOCK_NB))
return errno;
return mdbx_lck_op(lfd, F_SETLK, F_WRLCK, 0, 1);
int rc = mdbx_lck_op(lfd, F_SETLK, F_WRLCK, 0, 1);
if (rc != 0 && fallback2shared) {
while (flock(lfd, LOCK_SH)) {
int rc = errno;
if (rc != EINTR)
return rc;
}
}
return rc;
}
static __inline int mdbx_lck_shared(int lfd) {
@@ -89,8 +97,6 @@ int mdbx_lck_downgrade(MDBX_env *env, bool complete) {
return complete ? mdbx_lck_shared(env->me_lfd) : MDBX_SUCCESS;
}
int mdbx_lck_upgrade(MDBX_env *env) { return mdbx_lck_exclusive(env->me_lfd); }
int mdbx_rpid_set(MDBX_env *env) {
return mdbx_lck_op(env->me_lfd, F_SETLK, F_WRLCK, env->me_pid, 1);
}
@@ -159,7 +165,7 @@ bailout:
void mdbx_lck_destroy(MDBX_env *env) {
if (env->me_lfd != INVALID_HANDLE_VALUE) {
/* try get exclusive access */
if (env->me_lck && mdbx_lck_exclusive(env->me_lfd) == 0) {
if (env->me_lck && mdbx_lck_exclusive(env->me_lfd, false) == 0) {
mdbx_info("%s: got exclusive, drown mutexes", mdbx_func_);
int rc = pthread_mutex_destroy(&env->me_lck->mti_rmutex);
if (rc == 0)
@@ -227,7 +233,7 @@ static int internal_seize_lck(int lfd) {
assert(lfd != INVALID_HANDLE_VALUE);
/* try exclusive access */
int rc = mdbx_lck_exclusive(lfd);
int rc = mdbx_lck_exclusive(lfd, false);
if (rc == 0)
/* got exclusive */
return MDBX_RESULT_TRUE;
@@ -236,7 +242,7 @@ static int internal_seize_lck(int lfd) {
rc = mdbx_lck_shared(lfd);
if (rc == 0) {
/* got shared, try exclusive again */
rc = mdbx_lck_exclusive(lfd);
rc = mdbx_lck_exclusive(lfd, true);
if (rc == 0)
/* now got exclusive */
return MDBX_RESULT_TRUE;

View File

@@ -190,7 +190,9 @@ static int suspend_and_append(mdbx_handle_array_t **array,
(limit * 2 - ARRAY_LENGTH((*array)->handles)));
if (!ptr)
return MDBX_ENOMEM;
(*array) = (mdbx_handle_array_t *)ptr;
if (limit == ARRAY_LENGTH((*array)->handles))
memcpy(ptr, *array, sizeof(mdbx_handle_array_t));
*array = (mdbx_handle_array_t *)ptr;
(*array)->limit = limit * 2;
}
@@ -439,47 +441,6 @@ int mdbx_lck_downgrade(MDBX_env *env, bool complete) {
return MDBX_SUCCESS /* 7) now at S-? (used), done */;
}
int mdbx_lck_upgrade(MDBX_env *env) {
/* Transite from locked state (S-E) to exclusive-write (E-E) */
assert(env->me_fd != INVALID_HANDLE_VALUE);
assert(env->me_lfd != INVALID_HANDLE_VALUE);
/* 1) must be at S-E (locked), transite to ?_E (middle) */
if (!funlock(env->me_lfd, LCK_LOWER))
mdbx_panic("%s(%s) failed: errcode %u", mdbx_func_,
"S-E(locked) >> ?-E(middle)", GetLastError());
/* 3) now on ?-E (middle), try E-E (exclusive-write) */
mdbx_jitter4testing(false);
if (flock(env->me_lfd, LCK_EXCLUSIVE | LCK_DONTWAIT, LCK_LOWER))
return MDBX_RESULT_TRUE; /* 4) got E-E (exclusive-write), done */
/* 5) still on ?-E (middle) */
int rc = GetLastError();
mdbx_jitter4testing(false);
if (rc != ERROR_SHARING_VIOLATION && rc != ERROR_LOCK_VIOLATION) {
/* 6) something went wrong, report but continue */
mdbx_error("%s(%s) failed: errcode %u", mdbx_func_,
"?-E(middle) >> E-E(exclusive-write)", rc);
}
/* 7) still on ?-E (middle), try restore S-E (locked) */
mdbx_jitter4testing(false);
rc = flock(env->me_lfd, LCK_SHARED | LCK_DONTWAIT, LCK_LOWER)
? MDBX_RESULT_FALSE
: GetLastError();
mdbx_jitter4testing(false);
if (rc != MDBX_RESULT_FALSE) {
mdbx_fatal("%s(%s) failed: errcode %u", mdbx_func_,
"?-E(middle) >> S-E(locked)", rc);
return rc;
}
/* 8) now on S-E (locked) */
return MDBX_RESULT_FALSE;
}
void mdbx_lck_destroy(MDBX_env *env) {
int rc;

1232
src/mdbx.c

File diff suppressed because it is too large Load Diff

View File

@@ -548,7 +548,6 @@ int mdbx_lck_init(MDBX_env *env);
int mdbx_lck_seize(MDBX_env *env);
int mdbx_lck_downgrade(MDBX_env *env, bool complete);
int mdbx_lck_upgrade(MDBX_env *env);
void mdbx_lck_destroy(MDBX_env *env);
int mdbx_rdt_lock(MDBX_env *env);

View File

@@ -1,4 +1,4 @@
/* mdbx_chk.c - memory-mapped database check tool */
/* mdbx_chk.c - memory-mapped database check tool */
/*
* Copyright 2015-2018 Leonid Yuriev <leo@yuriev.ru>
@@ -61,12 +61,18 @@ static void signal_handler(int sig) {
#define EXIT_FAILURE_CHECK_MAJOR (EXIT_FAILURE + 1)
#define EXIT_FAILURE_CHECK_MINOR EXIT_FAILURE
typedef struct {
const char *name;
struct {
uint64_t total;
uint64_t empty;
} pages;
uint64_t payload_bytes;
uint64_t lost_bytes;
} walk_dbi_t;
struct {
const char *dbi_names[MAX_DBI];
uint64_t dbi_pages[MAX_DBI];
uint64_t dbi_empty_pages[MAX_DBI];
uint64_t dbi_payload_bytes[MAX_DBI];
uint64_t dbi_lost_bytes[MAX_DBI];
walk_dbi_t dbi[MAX_DBI];
short *pagemap;
uint64_t total_payload_bytes;
uint64_t pgcount;
@@ -95,7 +101,7 @@ struct problem *problems_list;
uint64_t total_problems;
static void
#ifdef __GNU__
#ifdef __GNUC__
__attribute__((format(printf, 1, 2)))
#endif
print(const char *msg, ...) {
@@ -110,7 +116,7 @@ static void
}
static void
#ifdef __GNU__
#ifdef __GNUC__
__attribute__((format(printf, 1, 2)))
#endif
error(const char *msg, ...) {
@@ -131,9 +137,9 @@ static void pagemap_cleanup(void) {
int i;
for (i = 1; i < MAX_DBI; ++i) {
if (walk.dbi_names[i]) {
free((void *)walk.dbi_names[i]);
walk.dbi_names[i] = NULL;
if (walk.dbi[i].name) {
free((void *)walk.dbi[i].name);
walk.dbi[i].name = NULL;
}
}
@@ -141,32 +147,35 @@ static void pagemap_cleanup(void) {
walk.pagemap = NULL;
}
static int pagemap_lookup_dbi(const char *dbi) {
static int last;
int i;
static walk_dbi_t *pagemap_lookup_dbi(const char *dbi_name) {
static walk_dbi_t *last;
if (last > 0 && strcmp(walk.dbi_names[last], dbi) == 0)
if (last && strcmp(last->name, dbi_name) == 0)
return last;
for (i = 1; walk.dbi_names[i] && last < MAX_DBI; ++i)
if (strcmp(walk.dbi_names[i], dbi) == 0)
return last = i;
if (i == MAX_DBI)
return -1;
walk.dbi_names[i] = strdup(dbi);
walk_dbi_t *dbi = walk.dbi + 1;
while (dbi->name) {
if (strcmp(dbi->name, dbi_name) == 0)
return last = dbi;
if (++dbi == walk.dbi + MAX_DBI)
return NULL;
}
dbi->name = strdup(dbi_name);
if (verbose > 1) {
print(" - found '%s' area\n", dbi);
print(" - found '%s' area\n", dbi_name);
fflush(NULL);
}
return last = i;
return last = dbi;
}
static void problem_add(const char *object, uint64_t entry_number,
const char *msg, const char *extra, ...) {
static void
#ifdef __GNUC__
__attribute__((format(printf, 4, 5)))
#endif
problem_add(const char *object, uint64_t entry_number, const char *msg,
const char *extra, ...) {
total_problems++;
if (!quiet) {
@@ -233,7 +242,7 @@ static uint64_t problems_pop(struct problem *list) {
}
static int pgvisitor(uint64_t pgno, unsigned pgnumber, void *ctx,
const char *dbi, const char *type, size_t nentries,
const char *dbi_name, const char *type, size_t nentries,
size_t payload_bytes, size_t header_bytes,
size_t unused_bytes) {
(void)ctx;
@@ -241,54 +250,58 @@ static int pgvisitor(uint64_t pgno, unsigned pgnumber, void *ctx,
if (type) {
uint64_t page_bytes = payload_bytes + header_bytes + unused_bytes;
size_t page_size = (size_t)pgnumber * envstat.ms_psize;
int index = pagemap_lookup_dbi(dbi);
if (index < 0)
walk_dbi_t *dbi = pagemap_lookup_dbi(dbi_name);
if (!dbi)
return MDBX_ENOMEM;
if (verbose > 2 && (!only_subdb || strcmp(only_subdb, dbi) == 0)) {
if (verbose > 2 && (!only_subdb || strcmp(only_subdb, dbi_name) == 0)) {
if (pgnumber == 1)
print(" %s-page %" PRIu64, type, pgno);
else
print(" %s-span %" PRIu64 "[%u]", type, pgno, pgnumber);
print(" of %s: header %" PRIiPTR ", payload %" PRIiPTR
", unused %" PRIiPTR "\n",
dbi, header_bytes, payload_bytes, unused_bytes);
dbi_name, header_bytes, payload_bytes, unused_bytes);
}
walk.pgcount += pgnumber;
if (unused_bytes > page_size)
problem_add("page", pgno, "illegal unused-bytes", "%u < %i < %u", 0,
unused_bytes, envstat.ms_psize);
problem_add("page", pgno, "illegal unused-bytes",
"%u < %" PRIuPTR " < %u", 0, unused_bytes, envstat.ms_psize);
if (header_bytes < (int)sizeof(long) ||
(size_t)header_bytes >= envstat.ms_psize - sizeof(long))
problem_add("page", pgno, "illegal header-length",
"%" PRIuPTR " < %i < %" PRIuPTR "", sizeof(long),
"%" PRIuPTR " < %" PRIuPTR " < %" PRIuPTR, sizeof(long),
header_bytes, envstat.ms_psize - sizeof(long));
if (payload_bytes < 1) {
if (nentries > 1) {
problem_add("page", pgno, "zero size-of-entry",
"payload %i bytes, %i entries", payload_bytes, nentries);
"payload %" PRIuPTR " bytes, %" PRIuPTR " entries",
payload_bytes, nentries);
if ((size_t)header_bytes + unused_bytes < page_size) {
/* LY: hush a misuse error */
page_bytes = page_size;
}
} else {
problem_add("page", pgno, "empty", "payload %i bytes, %i entries",
problem_add("page", pgno, "empty",
"payload %" PRIuPTR " bytes, %" PRIuPTR " entries",
payload_bytes, nentries);
walk.dbi_empty_pages[index] += 1;
dbi->pages.empty += 1;
}
}
if (page_bytes != page_size) {
problem_add("page", pgno, "misused",
"%" PRIu64 " != %" PRIu64 " (%ih + %ip + %iu)", page_size,
page_bytes, header_bytes, payload_bytes, unused_bytes);
"%" PRIu64 " != %" PRIu64 " (%" PRIuPTR "h + %" PRIuPTR
"p + %" PRIuPTR "u)",
page_size, page_bytes, header_bytes, payload_bytes,
unused_bytes);
if (page_size > page_bytes)
walk.dbi_lost_bytes[index] += page_size - page_bytes;
dbi->lost_bytes += page_size - page_bytes;
} else {
walk.dbi_payload_bytes[index] += payload_bytes + header_bytes;
dbi->payload_bytes += payload_bytes + header_bytes;
walk.total_payload_bytes += payload_bytes + header_bytes;
}
@@ -299,10 +312,10 @@ static int pgvisitor(uint64_t pgno, unsigned pgnumber, void *ctx,
"%" PRIu64 " > %" PRIu64 "", pgno, lastpgno);
else if (walk.pagemap[pgno])
problem_add("page", pgno, "already used", "in %s",
walk.dbi_names[walk.pagemap[pgno]]);
walk.dbi[walk.pagemap[pgno]].name);
else {
walk.pagemap[pgno] = (short)index;
walk.dbi_pages[index] += 1;
walk.pagemap[pgno] = (short)(dbi - walk.dbi);
dbi->pages.total += 1;
}
++pgno;
} while (--pgnumber);
@@ -337,16 +350,22 @@ static int handle_freedb(const uint64_t record_number, const MDBX_val *key,
problem_add("entry", record_number, "wrong txn-id", "%" PRIaTXN "", txnid);
if (data->iov_len < sizeof(pgno_t) || data->iov_len % sizeof(pgno_t))
problem_add("entry", record_number, "wrong idl size", "%" PRIuPTR "",
problem_add("entry", txnid, "wrong idl size", "%" PRIuPTR "",
data->iov_len);
else {
const pgno_t number = *iptr++;
if (number >= MDBX_PNL_UM_MAX)
problem_add("entry", record_number, "wrong idl length", "%" PRIiPTR "",
number);
else if ((number + 1) * sizeof(pgno_t) != data->iov_len)
problem_add("entry", record_number, "mismatch idl length",
"%" PRIuSIZE " != %" PRIuSIZE "",
if (number < 1 || number >= INT_MAX / 2)
problem_add("entry", txnid, "wrong idl length", "%" PRIaPGNO, number);
else if ((number + 1) * sizeof(pgno_t) > data->iov_len)
problem_add("entry", txnid, "trimmed idl",
"%" PRIuSIZE " > %" PRIuSIZE " (corruption)",
(number + 1) * sizeof(pgno_t), data->iov_len);
else if (data->iov_len - (number + 1) * sizeof(pgno_t) >=
/* LY: allow gap upto one page. it is ok
* and better than shink-and-retry inside mdbx_update_gc() */
envstat.ms_psize)
problem_add("entry", txnid, "extra idl space",
"%" PRIuSIZE " < %" PRIuSIZE " (minor, not a trouble)",
(number + 1) * sizeof(pgno_t), data->iov_len);
else {
freedb_pages += number;
@@ -359,12 +378,12 @@ static int handle_freedb(const uint64_t record_number, const MDBX_val *key,
for (unsigned i = 0; i < number; ++i) {
const pgno_t pg = iptr[i];
if (pg < NUM_METAS || pg > envinfo.mi_last_pgno)
problem_add("entry", record_number, "wrong idl entry",
problem_add("entry", txnid, "wrong idl entry",
"%u < %" PRIaPGNO " < %" PRIu64 "", NUM_METAS, pg,
envinfo.mi_last_pgno);
else if (MDBX_PNL_DISORDERED(prev, pg)) {
bad = " [bad sequence]";
problem_add("entry", record_number, "bad sequence",
problem_add("entry", txnid, "bad sequence",
"%" PRIaPGNO " <> %" PRIaPGNO "", prev, pg);
}
prev = pg;
@@ -516,7 +535,7 @@ static int process_db(MDBX_dbi dbi, char *name, visitor *handler, bool silent) {
if (key.iov_len > maxkeysize) {
problem_add("entry", record_count, "key length exceeds max-key-size",
"%" PRIuPTR " > %u", key.iov_len, maxkeysize);
"%" PRIuPTR " > %" PRIuPTR, key.iov_len, maxkeysize);
} else if ((flags & MDBX_INTEGERKEY) && key.iov_len != sizeof(uint64_t) &&
key.iov_len != sizeof(uint32_t)) {
problem_add("entry", record_count, "wrong key length",
@@ -758,7 +777,7 @@ static void print_size(const char *prefix, const uint64_t value,
}
int main(int argc, char *argv[]) {
int i, rc;
int rc;
char *prog = argv[0];
char *envname;
int problems_maindb = 0, problems_freedb = 0, problems_meta = 0;
@@ -778,14 +797,14 @@ int main(int argc, char *argv[]) {
}
#endif
walk.dbi_names[0] = "@gc";
walk.dbi[FREE_DBI].name = "@gc";
atexit(pagemap_cleanup);
if (argc < 2) {
usage(prog);
}
while ((i = getopt(argc, argv, "Vvqnwcds:")) != EOF) {
for (int i; (i = getopt(argc, argv, "Vvqnwcds:")) != EOF;) {
switch (i) {
case 'V':
printf("%s (%s, build %s)\n", mdbx_version.git.describe,
@@ -988,24 +1007,25 @@ int main(int argc, char *argv[]) {
goto bailout;
}
uint64_t n;
for (n = 0; n < lastpgno; ++n)
for (uint64_t n = 0; n < lastpgno; ++n)
if (!walk.pagemap[n])
walk.dbi_pages[0] += 1;
walk.dbi[FREE_DBI].pages.total += 1;
empty_pages = lost_bytes = 0;
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
empty_pages += walk.dbi_empty_pages[i];
lost_bytes += walk.dbi_lost_bytes[i];
for (walk_dbi_t *dbi = walk.dbi; ++dbi < walk.dbi + MAX_DBI && dbi->name;) {
empty_pages += dbi->pages.empty;
lost_bytes += dbi->lost_bytes;
}
if (verbose) {
uint64_t total_page_bytes = walk.pgcount * envstat.ms_psize;
print(" - dbi pages: %" PRIu64 " total", walk.pgcount);
if (verbose > 1)
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i)
print(", %s %" PRIu64 "", walk.dbi_names[i], walk.dbi_pages[i]);
print(", %s %" PRIu64 "\n", walk.dbi_names[0], walk.dbi_pages[0]);
for (walk_dbi_t *dbi = walk.dbi;
++dbi < walk.dbi + MAX_DBI && dbi->name;)
print(", %s %" PRIu64, dbi->name, dbi->pages.total);
print(", %s %" PRIu64 "\n", walk.dbi[FREE_DBI].name,
walk.dbi[FREE_DBI].pages.total);
if (verbose > 1) {
print(" - space info: total %" PRIu64 " bytes, payload %" PRIu64
" (%.1f%%), unused "
@@ -1015,19 +1035,19 @@ int main(int argc, char *argv[]) {
total_page_bytes - walk.total_payload_bytes,
(total_page_bytes - walk.total_payload_bytes) * 100.0 /
total_page_bytes);
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
uint64_t dbi_bytes = walk.dbi_pages[i] * envstat.ms_psize;
for (walk_dbi_t *dbi = walk.dbi;
++dbi < walk.dbi + MAX_DBI && dbi->name;) {
uint64_t dbi_bytes = dbi->pages.total * envstat.ms_psize;
print(" %s: subtotal %" PRIu64 " bytes (%.1f%%),"
" payload %" PRIu64 " (%.1f%%), unused %" PRIu64 " (%.1f%%)",
walk.dbi_names[i], dbi_bytes,
dbi_bytes * 100.0 / total_page_bytes, walk.dbi_payload_bytes[i],
walk.dbi_payload_bytes[i] * 100.0 / dbi_bytes,
dbi_bytes - walk.dbi_payload_bytes[i],
(dbi_bytes - walk.dbi_payload_bytes[i]) * 100.0 / dbi_bytes);
if (walk.dbi_empty_pages[i])
print(", %" PRIu64 " empty pages", walk.dbi_empty_pages[i]);
if (walk.dbi_lost_bytes[i])
print(", %" PRIu64 " bytes lost", walk.dbi_lost_bytes[i]);
dbi->name, dbi_bytes, dbi_bytes * 100.0 / total_page_bytes,
dbi->payload_bytes, dbi->payload_bytes * 100.0 / dbi_bytes,
dbi_bytes - dbi->payload_bytes,
(dbi_bytes - dbi->payload_bytes) * 100.0 / dbi_bytes);
if (dbi->pages.empty)
print(", %" PRIu64 " empty pages", dbi->pages.empty);
if (dbi->lost_bytes)
print(", %" PRIu64 " bytes lost", dbi->lost_bytes);
print("\n");
}
}
@@ -1084,9 +1104,9 @@ int main(int argc, char *argv[]) {
error("used pages mismatch (%" PRIu64 " != %" PRIu64 ")\n",
walk.pgcount, lastpgno - freedb_pages);
}
if (walk.dbi_pages[0] != freedb_pages) {
if (walk.dbi[FREE_DBI].pages.total != freedb_pages) {
error("gc pages mismatch (%" PRIu64 " != %" PRIu64 ")\n",
walk.dbi_pages[0], freedb_pages);
walk.dbi[FREE_DBI].pages.total, freedb_pages);
}
} else if (verbose) {
print(" - skip check used and gc pages (btree-traversal with "

View File

@@ -18,7 +18,7 @@
#error "API version mismatch!"
#endif
#define MDBX_VERSION_RELEASE 5
#define MDBX_VERSION_RELEASE 6
#define MDBX_VERSION_REVISION 1
/*LIBMDBX_EXPORTS*/ const mdbx_version_info mdbx_version = {

View File

@@ -43,6 +43,11 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
if (narg + 1 < argc && strncmp("--", argv[narg + 1], 2) != 0) {
*value = argv[narg + 1];
if (strcmp(*value, "default") == 0) {
if (!default_value)
failure("Option '--%s' doen't accept default value\n", option);
*value = default_value;
}
++narg;
return true;
}
@@ -57,9 +62,15 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty) {
return parse_option(argc, argv, narg, option, value, allow_empty,
allow_empty ? "" : nullptr);
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty,
const char *default_value) {
const char *value_cstr;
if (!parse_option(argc, argv, narg, option, &value_cstr,
allow_empty ? "" : nullptr))
if (!parse_option(argc, argv, narg, option, &value_cstr, default_value))
return false;
if (!allow_empty && strlen(value_cstr) == 0)
@@ -75,7 +86,7 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
if (!parse_option(argc, argv, narg, option, &list))
return false;
mask = 0;
unsigned clear = 0;
while (*list) {
if (*list == ',' || *list == ' ' || *list == '\t') {
++list;
@@ -83,14 +94,21 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
}
const char *const comma = strchr(list, ',');
const bool strikethrough = *list == '-' || *list == '~';
if (strikethrough || *list == '+')
++list;
else
mask = clear;
const size_t len = (comma) ? comma - list : strlen(list);
const option_verb *scan = verbs;
while (true) {
if (!scan->verb)
failure("Unknown verb '%.*s', for option '==%s'\n", (int)len, list,
option);
if (strlen(scan->verb) == len && strncmp(list, scan->verb, len) == 0) {
mask |= scan->mask;
mask = strikethrough ? mask & ~scan->mask : mask | scan->mask;
clear = strikethrough ? clear & ~scan->mask : clear | scan->mask;
list += len;
break;
}
@@ -103,15 +121,36 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint64_t &value, const scale_mode scale,
const uint64_t minval, const uint64_t maxval) {
const uint64_t minval, const uint64_t maxval,
const uint64_t default_value) {
const char *value_cstr;
if (!parse_option(argc, argv, narg, option, &value_cstr))
return false;
if (default_value && strcmp(value_cstr, "default") == 0) {
value = default_value;
return true;
}
if (strcmp(value_cstr, "min") == 0 || strcmp(value_cstr, "minimal") == 0) {
value = minval;
return true;
}
if (strcmp(value_cstr, "max") == 0 || strcmp(value_cstr, "maximal") == 0) {
value = maxval;
return true;
}
char *suffix = nullptr;
errno = 0;
unsigned long raw = strtoul(value_cstr, &suffix, 0);
unsigned long long raw = strtoull(value_cstr, &suffix, 0);
if ((suffix && *suffix) || errno) {
suffix = nullptr;
errno = 0;
raw = strtoull(value_cstr, &suffix, 10);
}
if (errno)
failure("Option '--%s' expects a numeric value (%s)\n", option,
test_strerror(errno));
@@ -167,28 +206,58 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
unsigned &value, const scale_mode scale,
const unsigned minval, const unsigned maxval) {
const unsigned minval, const unsigned maxval,
const unsigned default_value) {
uint64_t huge;
if (!parse_option(argc, argv, narg, option, huge, scale, minval, maxval))
if (!parse_option(argc, argv, narg, option, huge, scale, minval, maxval,
default_value))
return false;
value = (unsigned)huge;
return true;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint8_t &value, const uint8_t minval, const uint8_t maxval) {
uint8_t &value, const uint8_t minval, const uint8_t maxval,
const uint8_t default_value) {
uint64_t huge;
if (!parse_option(argc, argv, narg, option, huge, no_scale, minval, maxval))
if (!parse_option(argc, argv, narg, option, huge, no_scale, minval, maxval,
default_value))
return false;
value = (uint8_t)huge;
return true;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int64_t &value, const int64_t minval, const int64_t maxval,
const int64_t default_value) {
uint64_t proxy = (uint64_t)value;
if (parse_option(argc, argv, narg, option, proxy, config::binary,
(uint64_t)minval, (uint64_t)maxval,
(uint64_t)default_value)) {
value = (int64_t)proxy;
return true;
}
return false;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int32_t &value, const int32_t minval, const int32_t maxval,
const int32_t default_value) {
uint64_t proxy = (uint64_t)value;
if (parse_option(argc, argv, narg, option, proxy, config::binary,
(uint64_t)minval, (uint64_t)maxval,
(uint64_t)default_value)) {
value = (int32_t)proxy;
return true;
}
return false;
}
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool &value) {
const char *value_cstr = NULL;
const char *value_cstr = nullptr;
if (!parse_option(argc, argv, narg, option, &value_cstr, "yes")) {
const char *current = argv[narg];
if (strncmp(current, "--no-", 5) == 0 && strcmp(current + 5, option) == 0) {
@@ -257,7 +326,7 @@ static void dump_verbs(const char *caption, size_t bits,
++verbs;
}
logging::feed("\n");
logging::feed("%s\n", (*comma == '\0') ? "none" : "");
}
static void dump_duration(const char *caption, unsigned duration) {
@@ -288,8 +357,12 @@ void dump(const char *title) {
: i->params.pathname_log.c_str());
}
log_info("database: %s, size %" PRIu64 "\n", i->params.pathname_db.c_str(),
i->params.size);
log_info("database: %s, size %" PRIuPTR "[%" PRIiPTR "..%" PRIiPTR
", %i %i, %i]\n",
i->params.pathname_db.c_str(), i->params.size_now,
i->params.size_lower, i->params.size_upper,
i->params.shrink_threshold, i->params.growth_step,
i->params.pagesize);
dump_verbs("mode", i->params.mode_flags, mode_bits);
dump_verbs("table", i->params.table_flags, table_bits);
@@ -306,7 +379,13 @@ void dump(const char *title) {
log_info("threads %u\n", i->params.nthreads);
log_info("keygen.case: %s\n", keygencase2str(i->params.keygen.keycase));
log_info(
"keygen.params: case %s, width %u, mesh %u, rotate %u, offset %" PRIu64
", split %u/%u\n",
keygencase2str(i->params.keygen.keycase), i->params.keygen.width,
i->params.keygen.mesh, i->params.keygen.rotate, i->params.keygen.offset,
i->params.keygen.split,
i->params.keygen.width - i->params.keygen.split);
log_info("keygen.seed: %u\n", i->params.keygen.seed);
log_info("key: minlen %u, maxlen %u\n", i->params.keylen_min,
i->params.keylen_max);
@@ -469,3 +548,27 @@ bool actor_config::deserialize(const char *str, actor_config &config) {
TRACE("<< actor_config::deserialize: OK\n");
return true;
}
unsigned actor_params::mdbx_keylen_min() const {
return (table_flags & MDBX_INTEGERKEY) ? 4 : 0;
}
unsigned actor_params::mdbx_keylen_max() const {
return (table_flags & MDBX_INTEGERKEY)
? 8
: std::min((unsigned)mdbx_get_maxkeysize(pagesize),
(unsigned)UINT16_MAX);
}
unsigned actor_params::mdbx_datalen_min() const {
return (table_flags & MDBX_INTEGERDUP) ? 4 : 0;
}
unsigned actor_params::mdbx_datalen_max() const {
return (table_flags & MDBX_INTEGERDUP)
? 8
: std::min((table_flags & MDBX_DUPSORT)
? (unsigned)mdbx_get_maxkeysize(pagesize)
: (unsigned)MDBX_MAXDATASIZE,
(unsigned)UINT16_MAX);
}

View File

@@ -62,6 +62,10 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty = false);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
std::string &value, bool allow_empty,
const char *default_value);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool &value);
@@ -75,16 +79,25 @@ bool parse_option(int argc, char *const argv[], int &narg, const char *option,
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint64_t &value, const scale_mode scale,
const uint64_t minval = 0, const uint64_t maxval = INT64_MAX);
const uint64_t minval = 0, const uint64_t maxval = INT64_MAX,
const uint64_t default_value = 0);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
unsigned &value, const scale_mode scale,
const unsigned minval = 0, const unsigned maxval = INT32_MAX);
const unsigned minval = 0, const unsigned maxval = INT32_MAX,
const unsigned default_value = 0);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
uint8_t &value, const uint8_t minval = 0,
const uint8_t maxval = 255);
const uint8_t maxval = 255, const uint8_t default_value = 0);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int64_t &value, const int64_t minval, const int64_t maxval,
const int64_t default_value = -1);
bool parse_option(int argc, char *const argv[], int &narg, const char *option,
int32_t &value, const int32_t minval, const int32_t maxval,
const int32_t default_value = -1);
//-----------------------------------------------------------------------------
#pragma pack(push, 1)
@@ -121,6 +134,8 @@ struct keygen_params_pod {
* Иначе говоря, нет смысла в со-координации генерации паттернов для
* ключей и значений. Более того, генерацию значений всегда необходимо
* рассматривать в контексте связки с одним значением ключа.
* - Тем не менее, во всех случаях достаточно важным является равномерная
* всех возможных сочетаний длин ключей и данных.
*
* width:
* Большинство тестов предполагают создание или итерирование некоторого
@@ -156,7 +171,7 @@ struct keygen_params_pod {
* псевдо-случайные значений ключей без псевдо-случайности в значениях.
*
* Такое ограничение соответствуют внутренней алгоритмике libmdbx. Проще
* говоря мы можем проверить движок псевдо-случайной последовательностью
* говоря, мы можем проверить движок псевдо-случайной последовательностью
* ключей на таблицах без дубликатов (без multi-value), а затем проверить
* корректность работу псевдо-случайной последовательностью значений на
* таблицах с дубликатами (с multi-value), опционально добавляя
@@ -203,7 +218,12 @@ struct actor_params_pod {
unsigned mode_flags;
unsigned table_flags;
uint64_t size;
intptr_t size_lower;
intptr_t size_now;
intptr_t size_upper;
int shrink_threshold;
int growth_step;
int pagesize;
unsigned test_duration;
unsigned test_nops;
@@ -246,6 +266,11 @@ struct actor_params : public config::actor_params_pod {
std::string pathname_log;
std::string pathname_db;
void set_defaults(const std::string &tmpdir);
unsigned mdbx_keylen_min() const;
unsigned mdbx_keylen_max() const;
unsigned mdbx_datalen_min() const;
unsigned mdbx_datalen_max() const;
};
struct actor_config : public config::actor_config_pod {

65
test/gc.sh Executable file
View File

@@ -0,0 +1,65 @@
#!/bin/bash
set -euo pipefail
make check
TESTDB_PREFIX=${1:-/dev/shm/mdbx-gc-test}.
function rep9 { printf "%*s" $1 '' | tr ' ' '9'; }
function join { local IFS="$1"; shift; echo "$*"; }
function bit2option { local -n arr=$1; (( ($2&(1<<$3)) != 0 )) && echo -n '+' || echo -n '-'; echo "${arr[$3]}"; }
options=(writemap coalesce lifo)
function bits2list {
local -n arr=$1
local i
local list=()
for ((i=0; i<${#arr[@]}; ++i)) do
list[$i]=$(bit2option $1 $2 $i)
done
join , "${list[@]}"
}
function probe {
echo "=============================================== $(date)"
echo "${caption}: $*"
rm -f ${TESTDB_PREFIX}* \
&& ./mdbx_test --pathname=${TESTDB_PREFIX}db "$@" | lz4 > ${TESTDB_PREFIX}log.lz4 \
&& ./mdbx_chk -nvv ${TESTDB_PREFIX}db | tee ${TESTDB_PREFIX}chk \
|| (echo "FAILED"; exit 1)
}
###############################################################################
count=0
for nops in {2..7}; do
for ((wbatch=nops-1; wbatch > 0; --wbatch)); do
loops=$(((3333 >> nops) / nops + 1))
for ((rep=0; rep++ < loops; )); do
for ((bits=2**${#options[@]}; --bits >= 0; )); do
seed=$(date +%N)
caption="Probe #$((++count)) int-key,w/o-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+key.integer,-data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) int-key,with-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+key.integer,+data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) int-key,int-data, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+key.integer,+data.integer --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} basic
caption="Probe #$((++count)) w/o-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=-data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=1111 \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} --hill
caption="Probe #$((++count)) with-dups, repeat ${rep} of ${loops}" probe \
--pagesize=min --size=6G --table=+data.dups --keylen.min=min --keylen.max=max --datalen.min=min --datalen.max=max \
--nops=$( rep9 $nops ) --batch.write=$( rep9 $wbatch ) --mode=$(bits2list options $bits) \
--keygen.seed=${seed} --hill
done
done
done
done
echo "=== ALL DONE ====================== $(date)"

View File

@@ -53,7 +53,7 @@ bool testcase_hill::run() {
*/
/* TODO: работа в несколько потоков */
keyvalue_maker.setup(config.params, 0 /* thread_number */);
keyvalue_maker.setup(config.params, config.actor_id, 0 /* thread_number */);
keygen::buffer a_key = keygen::alloc(config.params.keylen_max);
keygen::buffer a_data_0 = keygen::alloc(config.params.datalen_max);
@@ -65,7 +65,9 @@ bool testcase_hill::run() {
? MDBX_NODUPDATA
: MDBX_NODUPDATA | MDBX_NOOVERWRITE;
const unsigned update_flags =
MDBX_CURRENT | MDBX_NODUPDATA | MDBX_NOOVERWRITE;
(config.params.table_flags & MDBX_DUPSORT)
? MDBX_CURRENT | MDBX_NODUPDATA | MDBX_NOOVERWRITE
: MDBX_NODUPDATA;
uint64_t serial_count = 0;
unsigned txn_nops = 0;
@@ -115,7 +117,7 @@ bool testcase_hill::run() {
rc = mdbx_replace(txn_guard.get(), dbi, &a_key->value, &a_data_0->value,
&a_data_1->value, update_flags);
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_put(update-a: 1->0)", rc);
failure_perror("mdbx_replace(update-a: 1->0)", rc);
if (++txn_nops >= config.params.batch_write) {
txn_restart(false, false);
@@ -156,8 +158,6 @@ bool testcase_hill::run() {
a_serial);
generate_pair(a_serial, a_key, a_data_0, 0);
generate_pair(a_serial, a_key, a_data_1, age_shift);
if (a_serial == 808)
log_trace("!!!");
int rc = mdbx_replace(txn_guard.get(), dbi, &a_key->value, &a_data_1->value,
&a_data_0->value, update_flags);
if (unlikely(rc != MDBX_SUCCESS))

View File

@@ -58,7 +58,11 @@ bool testcase_jitter::run() {
jitter_delay();
db_close();
report(1);
/* just 'align' nops with other tests with batching */
const auto batching =
std::max(config.params.batch_read, config.params.batch_write);
report(std::max(1u, batching / 2));
}
return true;
}

View File

@@ -30,7 +30,7 @@ serial_t injective(const serial_t serial,
/* LY: All these "magic" prime numbers were found
* and verified with a bit of brute force. */
static const uint64_t m[64 - serial_minwith] = {
static const uint64_t m[64 - serial_minwith + 1] = {
/* 8 - 24 */
113, 157, 397, 653, 1753, 5641, 9697, 23873, 25693, 80833, 105953, 316937,
309277, 834497, 1499933, 4373441, 10184137,
@@ -43,26 +43,31 @@ serial_t injective(const serial_t serial,
2420886491930041, 3601632139991929, 11984491914483833, 21805846439714153,
23171543400565993, 53353226456762893, 155627817337932409,
227827205384840249, 816509268558278821, 576933057762605689,
2623957345935638441, 5048241705479929949, 4634245581946485653};
static const uint8_t s[64 - serial_minwith] = {
2623957345935638441, 5048241705479929949, 4634245581946485653,
4613509448041658233, 4952535426879925961};
static const uint8_t s[64 - serial_minwith + 1] = {
/* 8 - 24 */
2, 3, 4, 4, 2, 4, 3, 3, 7, 3, 3, 4, 8, 3, 10, 3, 11,
/* 25 - 64 */
11, 9, 9, 9, 11, 10, 5, 14, 11, 16, 14, 12, 13, 16, 19, 10, 10, 21, 7, 20,
10, 14, 22, 19, 3, 21, 18, 19, 26, 24, 2, 21, 25, 29, 24, 10, 11, 14};
10, 14, 22, 19, 3, 21, 18, 19, 26, 24, 2, 21, 25, 29, 24, 10, 11, 14, 20,
19};
serial_t result = serial * m[bits - 8];
const auto mult = m[bits - 8];
const auto shift = s[bits - 8];
serial_t result = serial * mult;
if (salt) {
const unsigned left = bits / 2;
const unsigned right = bits - left;
result = (result << left) | ((result & mask(bits)) >> right);
result = (result ^ salt) * m[bits - 8];
result = (result ^ salt) * mult;
}
result ^= result << s[bits - 8];
result ^= result << shift;
result &= mask(bits);
log_trace("keygen-injective: serial %" PRIu64 " into %" PRIu64, serial,
result);
log_trace("keygen-injective: serial %" PRIu64 "/%u @%" PRIx64 ",%u,%" PRIu64
" => %" PRIu64 "/%u",
serial, bits, mult, shift, salt, result, bits);
return result;
}
@@ -73,8 +78,9 @@ void __hot maker::pair(serial_t serial, const buffer &key, buffer &value,
assert(mapping.mesh <= mapping.width);
assert(mapping.rotate <= mapping.width);
assert(mapping.offset <= mask(mapping.width));
assert(!(key_essentials.flags & (MDBX_INTEGERDUP | MDBX_REVERSEDUP)));
assert(!(value_essentials.flags & (MDBX_INTEGERKEY | MDBX_REVERSEKEY)));
assert(!(key_essentials.flags &
~(MDBX_INTEGERKEY | MDBX_REVERSEKEY | MDBX_DUPSORT)));
assert(!(value_essentials.flags & ~(MDBX_INTEGERDUP | MDBX_REVERSEDUP)));
log_trace("keygen-pair: serial %" PRIu64 ", data-age %" PRIu64, serial,
value_age);
@@ -82,31 +88,49 @@ void __hot maker::pair(serial_t serial, const buffer &key, buffer &value,
if (mapping.mesh >= serial_minwith) {
serial =
(serial & ~mask(mapping.mesh)) | injective(serial, mapping.mesh, salt);
log_trace("keygen-pair: mesh %" PRIu64, serial);
log_trace("keygen-pair: mesh@%u => %" PRIu64, mapping.mesh, serial);
}
if (mapping.rotate) {
const unsigned right = mapping.rotate;
const unsigned left = mapping.width - right;
serial = (serial << left) | ((serial & mask(mapping.width)) >> right);
log_trace("keygen-pair: rotate %" PRIu64 ", 0x%" PRIx64, serial, serial);
log_trace("keygen-pair: rotate@%u => %" PRIu64 ", 0x%" PRIx64,
mapping.rotate, serial, serial);
}
serial = (serial + mapping.offset) & mask(mapping.width);
log_trace("keygen-pair: offset %" PRIu64, serial);
serial += base;
if (mapping.offset) {
serial = (serial + mapping.offset) & mask(mapping.width);
log_trace("keygen-pair: offset@%" PRIu64 " => %" PRIu64, mapping.offset,
serial);
}
if (base) {
serial += base;
log_trace("keygen-pair: base@%" PRIu64 " => %" PRIu64, base, serial);
}
serial_t key_serial = serial;
serial_t value_serial = value_age;
serial_t value_serial = value_age << mapping.split;
if (mapping.split) {
key_serial = serial >> mapping.split;
value_serial =
(serial & mask(mapping.split)) | (value_age << mapping.split);
if (key_essentials.flags & MDBX_DUPSORT) {
key_serial >>= mapping.split;
value_serial += serial & mask(mapping.split);
} else {
/* Без MDBX_DUPSORT требуется уникальность ключей, а для этого нельзя
* отбрасывать какие-либо биты serial после инъективного преобразования.
* Поэтому key_serial не трогаем, а в value_serial нелинейно вмешиваем
* запрошенное количество бит из serial */
value_serial +=
(serial ^ (serial >> mapping.split)) & mask(mapping.split);
}
value_serial |= value_age << mapping.split;
log_trace("keygen-pair: split@%u => k%" PRIu64 ", v%" PRIu64, mapping.split,
key_serial, value_serial);
}
log_trace("keygen-pair: key %" PRIu64 ", value %" PRIu64, key_serial,
value_serial);
mk(key_serial, key_essentials, *key);
mk(value_serial, value_essentials, *value);
@@ -118,26 +142,26 @@ void __hot maker::pair(serial_t serial, const buffer &key, buffer &value,
}
}
void maker::setup(const config::actor_params_pod &actor,
void maker::setup(const config::actor_params_pod &actor, unsigned actor_id,
unsigned thread_number) {
key_essentials.flags =
actor.table_flags & (MDBX_INTEGERKEY | MDBX_REVERSEKEY);
assert(actor.keylen_min < UINT8_MAX);
actor.table_flags & (MDBX_INTEGERKEY | MDBX_REVERSEKEY | MDBX_DUPSORT);
assert(actor.keylen_min <= UINT8_MAX);
key_essentials.minlen = (uint8_t)actor.keylen_min;
assert(actor.keylen_max < UINT16_MAX);
assert(actor.keylen_max <= UINT16_MAX);
key_essentials.maxlen = (uint16_t)actor.keylen_max;
value_essentials.flags =
actor.table_flags & (MDBX_INTEGERDUP | MDBX_REVERSEDUP);
assert(actor.datalen_min < UINT8_MAX);
assert(actor.datalen_min <= UINT8_MAX);
value_essentials.minlen = (uint8_t)actor.datalen_min;
assert(actor.datalen_max < UINT16_MAX);
assert(actor.datalen_max <= UINT16_MAX);
value_essentials.maxlen = (uint16_t)actor.datalen_max;
assert(thread_number < 2);
(void)thread_number;
mapping = actor.keygen;
salt = actor.keygen.seed * UINT64_C(14653293970879851569);
salt = (actor.keygen.seed + actor_id) * UINT64_C(14653293970879851569);
// FIXME: TODO
base = 0;
@@ -165,7 +189,7 @@ bool maker::increment(serial_t &serial, int delta) {
//-----------------------------------------------------------------------------
size_t length(serial_t serial) {
static size_t length(serial_t serial) {
size_t n = 0;
if (serial > UINT32_MAX) {
n = 4;
@@ -199,7 +223,10 @@ void __hot maker::mk(const serial_t serial, const essentials &params,
assert(params.maxlen >= length(serial));
out.value.iov_base = out.bytes;
out.value.iov_len = params.minlen;
out.value.iov_len =
(params.maxlen > params.minlen)
? params.minlen + serial % (params.maxlen - params.minlen)
: params.minlen;
if (params.flags & (MDBX_INTEGERKEY | MDBX_INTEGERDUP)) {
assert(params.maxlen == params.minlen);

View File

@@ -44,7 +44,7 @@ namespace keygen {
* - абсолютное значение ключей или разность между отдельными значениями;
*
* Соответственно, в общих чертах, схема генерации следующая:
* - вводится плоская одномерная "координата" uint64_t;
* - вводится плоская одномерная "координата" serial (uint64_t);
* - генерация специфических паттернов (последовательностей)
* реализуется посредством соответствующих преобразований "координат", при
* этом все подобные преобразования выполняются только над "координатой";
@@ -74,7 +74,7 @@ typedef uint64_t serial_t;
enum : serial_t {
serial_minwith = 8,
serial_maxwith = sizeof(serial_t) * 8,
serial_allones = ~(serial_t)0
serial_allones = ~(serial_t)0u
};
struct result {
@@ -85,6 +85,10 @@ struct result {
uint32_t u32;
uint64_t u64;
};
std::string as_string() const {
return std::string((const char *)value.iov_base, value.iov_len);
}
};
//-----------------------------------------------------------------------------
@@ -115,11 +119,10 @@ public:
void pair(serial_t serial, const buffer &key, buffer &value,
serial_t value_age);
void setup(const config::actor_params_pod &actor, unsigned thread_number);
void setup(const config::actor_params_pod &actor, unsigned actor_id,
unsigned thread_number);
bool increment(serial_t &serial, int delta);
};
size_t length(serial_t serial);
} /* namespace keygen */

View File

@@ -37,6 +37,31 @@ void __noreturn failure_perror(const char *what, int errnum) {
//-----------------------------------------------------------------------------
static void mdbx_logger(int type, const char *function, int line,
const char *msg, va_list args) {
logging::loglevel level = logging::info;
if (type & MDBX_DBG_EXTRA)
level = logging::extra;
if (type & MDBX_DBG_TRACE)
level = logging::trace;
if (type & MDBX_DBG_PRINT)
level = logging::verbose;
if (!function)
function = "unknown";
if (type & MDBX_DBG_ASSERT) {
log_error("mdbx: assertion failure: %s, %d", function, line);
level = logging::failure;
}
if (logging::output(
level,
strncmp(function, "mdbx_", 5) == 0 ? "%s: " : "mdbx: %s: ", function))
logging::feed_ap(msg, args);
if (type & MDBX_DBG_ASSERT)
abort();
}
namespace logging {
static std::string prefix;
@@ -44,8 +69,19 @@ static std::string suffix;
static loglevel level;
static FILE *last;
void setup(loglevel _level, const std::string &_prefix) {
void setlevel(loglevel _level) {
level = (_level > error) ? failure : _level;
int mdbx_dbg_opts = MDBX_DBG_ASSERT | MDBX_DBG_JITTER | MDBX_DBG_DUMP;
if (level <= trace)
mdbx_dbg_opts |= MDBX_DBG_TRACE;
if (level <= verbose)
mdbx_dbg_opts |= MDBX_DBG_PRINT;
int rc = mdbx_setup_debug(mdbx_dbg_opts, mdbx_logger);
log_trace("set mdbx debug-opts: 0x%02x", rc);
}
void setup(loglevel _level, const std::string &_prefix) {
setlevel(_level);
prefix = _prefix;
}
@@ -157,7 +193,7 @@ bool output(const logging::loglevel priority, const char *format, va_list ap) {
return true;
}
bool feed(const char *format, va_list ap) {
bool feed_ap(const char *format, va_list ap) {
if (!last)
return false;
@@ -176,7 +212,7 @@ bool feed(const char *format, ...) {
va_list ap;
va_start(ap, format);
feed(format, ap);
feed_ap(format, ap);
va_end(ap);
return true;
}

View File

@@ -46,11 +46,12 @@ enum loglevel {
const char *level2str(const loglevel level);
void setup(loglevel level, const std::string &prefix);
void setup(const std::string &prefix);
void setlevel(loglevel level);
bool output(const loglevel priority, const char *format, va_list ap);
bool __printf_args(2, 3)
output(const loglevel priority, const char *format, ...);
bool feed(const char *format, va_list ap);
bool feed_ap(const char *format, va_list ap);
bool __printf_args(1, 2) feed(const char *format, ...);
class local_suffix {

View File

@@ -1,4 +1,4 @@
/*
/*
* Copyright 2017-2018 Leonid Yuriev <leo@yuriev.ru>
* and other libmdbx authors: please see AUTHORS file.
* All rights reserved.
@@ -35,25 +35,31 @@ void actor_params::set_defaults(const std::string &tmpdir) {
mode_flags = MDBX_NOSUBDIR | MDBX_WRITEMAP | MDBX_MAPASYNC | MDBX_NORDAHEAD |
MDBX_NOMEMINIT | MDBX_COALESCE | MDBX_LIFORECLAIM;
table_flags = MDBX_DUPSORT;
size = 1024 * 1024 * 4;
size_lower = -1;
size_now = 1024 * 1024 * ((table_flags & MDBX_DUPSORT) ? 4 : 256);
size_upper = -1;
shrink_threshold = -1;
growth_step = -1;
pagesize = -1;
keygen.seed = 1;
keygen.keycase = kc_random;
keygen.width = 32;
keygen.mesh = 32;
keygen.width = (table_flags & MDBX_DUPSORT) ? 32 : 64;
keygen.mesh = keygen.width;
keygen.split = keygen.width / 2;
keygen.rotate = 0;
keygen.offset = 0;
keygen.rotate = 3;
keygen.offset = 41;
test_duration = 0;
test_nops = 1000;
nrepeat = 1;
nthreads = 1;
keylen_min = 0;
keylen_max = 42;
datalen_min = 0;
datalen_max = 256;
keylen_min = mdbx_keylen_min();
keylen_max = mdbx_keylen_max();
datalen_min = mdbx_datalen_min();
datalen_max = std::min(mdbx_datalen_max(), 256u * 1024 + 42);
batch_read = 4;
batch_write = 4;
@@ -148,10 +154,53 @@ int main(int argc, char *const argv[]) {
config::mode_bits))
continue;
if (config::parse_option(argc, argv, narg, "table", params.table_flags,
config::table_bits))
config::table_bits)) {
if ((params.table_flags & MDBX_DUPFIXED) == 0)
params.table_flags &= ~MDBX_INTEGERDUP;
if ((params.table_flags & MDBX_DUPSORT) == 0)
params.table_flags &=
~(MDBX_DUPFIXED | MDBX_REVERSEDUP | MDBX_INTEGERDUP);
continue;
if (config::parse_option(argc, argv, narg, "size", params.size,
config::binary, 4096 * 4))
}
if (config::parse_option(argc, argv, narg, "pagesize", params.pagesize,
mdbx_limits_pgsize_min(),
mdbx_limits_pgsize_max())) {
const unsigned keylen_max = params.mdbx_keylen_max();
if (params.keylen_min > keylen_max)
params.keylen_min = keylen_max;
if (params.keylen_max > keylen_max)
params.keylen_max = keylen_max;
const unsigned datalen_max = params.mdbx_datalen_max();
if (params.datalen_min > datalen_max)
params.datalen_min = datalen_max;
if (params.datalen_max > datalen_max)
params.datalen_max = datalen_max;
continue;
}
if (config::parse_option(argc, argv, narg, "size-lower", params.size_lower,
mdbx_limits_dbsize_min(params.pagesize),
mdbx_limits_dbsize_max(params.pagesize)))
continue;
if (config::parse_option(argc, argv, narg, "size", params.size_now,
mdbx_limits_dbsize_min(params.pagesize),
mdbx_limits_dbsize_max(params.pagesize)))
continue;
if (config::parse_option(argc, argv, narg, "size-upper", params.size_upper,
mdbx_limits_dbsize_min(params.pagesize),
mdbx_limits_dbsize_max(params.pagesize)))
continue;
if (config::parse_option(
argc, argv, narg, "shrink-threshold", params.shrink_threshold, 0,
(int)std::min((intptr_t)INT_MAX,
mdbx_limits_dbsize_max(params.pagesize) -
mdbx_limits_dbsize_min(params.pagesize))))
continue;
if (config::parse_option(
argc, argv, narg, "growth-step", params.growth_step, 0,
(int)std::min((intptr_t)INT_MAX,
mdbx_limits_dbsize_max(params.pagesize) -
mdbx_limits_dbsize_min(params.pagesize))))
continue;
if (config::parse_option(argc, argv, narg, "keygen.width",
@@ -188,20 +237,39 @@ int main(int argc, char *const argv[]) {
config::duration, 1))
continue;
if (config::parse_option(argc, argv, narg, "keylen.min", params.keylen_min,
config::no_scale, 0, params.keylen_max))
config::no_scale, params.mdbx_keylen_min(),
params.mdbx_keylen_max())) {
if ((params.table_flags & MDBX_INTEGERKEY) ||
params.keylen_max < params.keylen_min)
params.keylen_max = params.keylen_min;
continue;
}
if (config::parse_option(argc, argv, narg, "keylen.max", params.keylen_max,
config::no_scale, params.keylen_min,
mdbx_get_maxkeysize(0)))
config::no_scale, params.mdbx_keylen_min(),
params.mdbx_keylen_max())) {
if ((params.table_flags & MDBX_INTEGERKEY) ||
params.keylen_min > params.keylen_max)
params.keylen_min = params.keylen_max;
continue;
}
if (config::parse_option(argc, argv, narg, "datalen.min",
params.datalen_min, config::no_scale, 0,
params.datalen_max))
params.datalen_min, config::no_scale,
params.mdbx_datalen_min(),
params.mdbx_datalen_max())) {
if ((params.table_flags & MDBX_DUPFIXED) ||
params.datalen_max < params.datalen_min)
params.datalen_max = params.datalen_min;
continue;
}
if (config::parse_option(argc, argv, narg, "datalen.max",
params.datalen_max, config::no_scale,
params.datalen_min, MDBX_MAXDATASIZE))
params.mdbx_datalen_min(),
params.mdbx_datalen_max())) {
if ((params.table_flags & MDBX_DUPFIXED) ||
params.datalen_min > params.datalen_max)
params.datalen_min = params.datalen_max;
continue;
}
if (config::parse_option(argc, argv, narg, "batch.read", params.batch_read,
config::no_scale, 1))
continue;

View File

@@ -182,6 +182,9 @@ void osal_killall_actors(void) {
}
int osal_actor_poll(mdbx_pid_t &pid, unsigned timeout) {
struct timespec ts;
ts.tv_nsec = 0;
ts.tv_sec = timeout;
retry:
int status, options = WNOHANG;
#ifdef WUNTRACED
@@ -209,9 +212,16 @@ retry:
}
if (pid == 0) {
if (timeout && sleep(timeout))
/* child still running */
if (ts.tv_sec == 0 && ts.tv_nsec == 0)
ts.tv_nsec = 1;
if (nanosleep(&ts, &ts) == 0) {
/* timeout and no signal fomr child */
pid = 0;
return 0;
}
if (errno == EINTR)
goto retry;
return 0;
}
switch (errno) {

View File

@@ -53,7 +53,7 @@ void osal_wait4barrier(void) {
}
}
static HANDLE make_inharitable(HANDLE hHandle) {
static HANDLE make_inheritable(HANDLE hHandle) {
assert(hHandle != NULL && hHandle != INVALID_HANDLE_VALUE);
if (!DuplicateHandle(GetCurrentProcess(), hHandle, GetCurrentProcess(),
&hHandle, 0, TRUE,
@@ -71,7 +71,7 @@ void osal_setup(const std::vector<actor_config> &actors) {
HANDLE hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (!hEvent)
failure_perror("CreateEvent()", GetLastError());
hEvent = make_inharitable(hEvent);
hEvent = make_inheritable(hEvent);
log_trace("osal_setup: event %" PRIuPTR " -> %p", i, hEvent);
events[i] = hEvent;
}
@@ -79,12 +79,12 @@ void osal_setup(const std::vector<actor_config> &actors) {
hBarrierSemaphore = CreateSemaphore(NULL, 0, (LONG)actors.size(), NULL);
if (!hBarrierSemaphore)
failure_perror("CreateSemaphore(BarrierSemaphore)", GetLastError());
hBarrierSemaphore = make_inharitable(hBarrierSemaphore);
hBarrierSemaphore = make_inheritable(hBarrierSemaphore);
hBarrierEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (!hBarrierEvent)
failure_perror("CreateEvent(BarrierEvent)", GetLastError());
hBarrierEvent = make_inharitable(hBarrierEvent);
hBarrierEvent = make_inheritable(hBarrierEvent);
}
void osal_broadcast(unsigned id) {
@@ -262,15 +262,24 @@ int osal_actor_start(const actor_config &config, mdbx_pid_t &pid) {
STARTUPINFOA StartupInfo;
GetStartupInfoA(&StartupInfo);
char exename[_MAX_PATH];
char exename[_MAX_PATH + 1];
DWORD exename_size = sizeof(exename);
if (!QueryFullProcessImageNameA(GetCurrentProcess(), 0, exename,
&exename_size))
failure_perror("QueryFullProcessImageName()", GetLastError());
std::string cmdline = "test_mdbx.child ";
if (exename[1] != ':') {
exename_size = GetModuleFileName(NULL, exename, sizeof(exename));
if (exename_size >= sizeof(exename))
return ERROR_BAD_LENGTH;
}
std::string cmdline = "$ ";
ArgvQuote(cmdline, thunk_param(config));
if (cmdline.size() >= 32767)
return ERROR_BAD_LENGTH;
PROCESS_INFORMATION ProcessInformation;
if (!CreateProcessA(exename, const_cast<char *>(cmdline.c_str()),
NULL, // Retuned process handle is not inheritable.
@@ -280,7 +289,7 @@ int osal_actor_start(const actor_config &config, mdbx_pid_t &pid) {
NULL, // Inherit the parent's environment.
NULL, // Inherit the parent's current directory.
&StartupInfo, &ProcessInformation))
return GetLastError();
failure_perror(exename, GetLastError());
CloseHandle(ProcessInformation.hThread);
pid = ProcessInformation.dwProcessId;

View File

@@ -68,31 +68,6 @@ const char *keygencase2str(const keygen_case keycase) {
//-----------------------------------------------------------------------------
static void mdbx_logger(int type, const char *function, int line,
const char *msg, va_list args) {
logging::loglevel level = logging::info;
if (type & MDBX_DBG_EXTRA)
level = logging::extra;
if (type & MDBX_DBG_TRACE)
level = logging::trace;
if (type & MDBX_DBG_PRINT)
level = logging::verbose;
if (!function)
function = "unknown";
if (type & MDBX_DBG_ASSERT) {
log_error("mdbx: assertion failure: %s, %d", function, line);
level = logging::failure;
}
if (logging::output(
level,
strncmp(function, "mdbx_", 5) == 0 ? "%s: " : "mdbx: %s: ", function))
logging::feed(msg, args);
if (type & MDBX_DBG_ASSERT)
abort();
}
int testcase::oom_callback(MDBX_env *env, int pid, mdbx_tid_t tid, uint64_t txn,
unsigned gap, int retry) {
@@ -117,16 +92,8 @@ void testcase::db_prepare() {
log_trace(">> db_prepare");
assert(!db_guard);
int mdbx_dbg_opts = MDBX_DBG_ASSERT | MDBX_DBG_JITTER | MDBX_DBG_DUMP;
if (config.params.loglevel <= logging::trace)
mdbx_dbg_opts |= MDBX_DBG_TRACE;
if (config.params.loglevel <= logging::verbose)
mdbx_dbg_opts |= MDBX_DBG_PRINT;
int rc = mdbx_setup_debug(mdbx_dbg_opts, mdbx_logger);
log_trace("set mdbx debug-opts: 0x%02x", rc);
MDBX_env *env = nullptr;
rc = mdbx_env_create(&env);
int rc = mdbx_env_create(&env);
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_env_create()", rc);
@@ -149,7 +116,10 @@ void testcase::db_prepare() {
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_env_set_oomfunc()", rc);
rc = mdbx_env_set_mapsize(env, (size_t)config.params.size);
rc = mdbx_env_set_geometry(
env, config.params.size_lower, config.params.size_now,
config.params.size_upper, config.params.growth_step,
config.params.shrink_threshold, config.params.pagesize);
if (unlikely(rc != MDBX_SUCCESS))
failure_perror("mdbx_env_set_mapsize()", rc);

View File

@@ -78,21 +78,25 @@
<LinkIncremental>true</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<LinkIncremental>true</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<LinkIncremental>false</LinkIncremental>
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\</OutDir>
<IntDir>$(SolutionDir)$(Platform)\$(Configuration)\$(ProjectName)\</IntDir>
<TargetName>mdbx_test</TargetName>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>

View File

@@ -93,7 +93,7 @@ bool hex2data(const char *hex_begin, const char *hex_end, void *ptr,
//-----------------------------------------------------------------------------
/* TODO: replace my 'libmera' fomr t1ha. */
/* TODO: replace my 'libmera' from t1ha. */
uint64_t entropy_ticks(void) {
#if defined(EMSCRIPTEN)
return (uint64_t)emscripten_get_now();