ISAR/libmdbx
3
0
Fork 0
mirror of https://github.com/isar/libmdbx.git synced 2024-12-29 08:38:49 +08:00
Code Issues Packages Projects Releases Wiki Activity

mdbx: reformat and some cleanup (1/3 for rebirth).

Browse Source
This commit is contained in:
Leo Yuriev 2017-02-21 20:16:54 +03:00
parent e88adf3969
commit 2dc3e1ee5f
37 changed files with 16897 additions and 19073 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.travis.yml
View File

@ -12,4 +12,4 @@ compiler:
os:
- linux
script: if [ "${COVERITY_SCAN_BRANCH}" != 1 ]; then make all lmdb check; fi
script: if [ "${COVERITY_SCAN_BRANCH}" != 1 ]; then make all check; fi

223
CHANGES
View File

@ -1,223 +0,0 @@
MDBX
Add MDB_PREV_MULTIPLE
Add error MDB_PROBLEM, replace some MDB_CORRUPTED
Workarounds for glibc bugs: #21031 and 21032.
LMDB 0.9.20 Release Engineering
Fix mdb_load with escaped plaintext (ITS#8558)
Fix mdb_cursor_last / mdb_put interaction (ITS#8557)
LMDB 0.9.19 Release (2016/12/28)
Fix mdb_env_cwalk cursor init (ITS#8424)
Fix robust mutexes on Solaris 10/11 (ITS#8339)
Fix MDB_GET_BOTH on non-dup record (ITS#8393)
Optimize mdb_drop
Fix xcursors after mdb_cursor_del (ITS#8406)
Fix MDB_NEXT_DUP after mdb_cursor_del (ITS#8412)
Fix mdb_cursor_put resetting C_EOF (ITS#8489)
Fix mdb_env_copyfd2 to return EPIPE on SIGPIPE (ITS#8504)
Fix mdb_env_copy with empty DB (ITS#8209)
Fix behaviors with fork (ITS#8505)
Fix mdb_dbi_open with mainDB cursors (ITS#8542)
Fix F_NOCACHE on MacOS, error is non-fatal (ITS#7682)
Documentation
Cleanup doxygen nits
Note reserved vs actual mem/disk usage
LMDB 0.9.18 Release (2016/02/05)
already done for mdbx - Fix robust mutex detection on glibc 2.10-11 (ITS#8330)
Fix page_search_root assert on FreeDB (ITS#8336)
Fix MDB_APPENDDUP vs. rewrite(single item) (ITS#8334)
n/a for mdbx - Fix mdb_copy of large files on Windows
Fix subcursor move after delete (ITS#8355)
Fix mdb_midl_shrink off-by-one (ITS#8363)
n/a for mdbx - Check for utf8_to_utf16 failures (ITS#7992)
Catch strdup failure in mdb_dbi_open
Build
already done for mdbx - Additional makefile var tweaks (ITS#8169)
Documentation
Add Getting Started page
Update WRITEMAP description
LMDB 0.9.17 Release (2015/11/30)
Fix ITS#7377 catch calloc failure
Fix ITS#8237 regression from ITS#7589
Fix ITS#8238 page_split for DUPFIXED pages
Fix ITS#8221 MDB_PAGE_FULL on delete/rebalance
Fix ITS#8258 rebalance/split assert
Fix ITS#8263 cursor_put cursor tracking
Fix ITS#8264 cursor_del cursor tracking
Fix ITS#8310 cursor_del cursor tracking
Fix ITS#8299 mdb_del cursor tracking
Fix ITS#8300 mdb_del cursor tracking
Fix ITS#8304 mdb_del cursor tracking
Fix ITS#7771 fakepage cursor tracking
Fix ITS#7789 ensure mapsize >= pages in use
Fix ITS#7971 mdb_txn_renew0() new reader slots
already done for mdbx - Fix ITS#7969 use __sync_synchronize on non-x86
Fix ITS#8311 page_split from update_key
Fix ITS#8312 loose pages in nested txn
Fix ITS#8313 mdb_rebalance dummy cursor
Fix ITS#8315 dirty_room in nested txn
Fix ITS#8323 dirty_list in nested txn
Fix ITS#8316 page_merge cursor tracking
Fix ITS#8319 mdb_load error messages
Fix ITS#8320 mdb_load plaintext input
Fix ITS#8321 cursor tracking
Added mdb_txn_id() (ITS#7994)
Added robust mutex support
Miscellaneous cleanup/simplification
Build
Create install dirs if needed (ITS#8256)
not affected mdbx - Fix ThreadProc decl on Win32/MSVC (ITS#8270)
not affected mdbx - Added ssize_t typedef for MSVC (ITS#8067)
not affected mdbx - Use ANSI apis on Windows (ITS#8069)
already done for mdbx - Use O_SYNC if O_DSYNC,MDB_DSYNC are not defined (ITS#7209)
already done for mdbx - Allow passing AR to make (ITS#8168)
Allow passing mandir to make install (ITS#8169)
LMDB 0.9.16 Release (2015/08/14)
Fix cursor EOF bug (ITS#8190)
Fix handling of subDB records (ITS#8181)
Fix mdb_midl_shrink() usage (ITS#8200)
not affected mdbx - fix reference to EINTR on WIN32 from ITS#8106 (ITS#8192)
LMDB 0.9.15 Release (2015/06/19)
Fix txn init (ITS#7961,#7987)
Fix MDB_PREV_DUP (ITS#7955,#7671)
Fix compact of empty env (ITS#7956)
Fix mdb_copy file mode
Fix mdb_env_close() after failed mdb_env_open()
Fix mdb_rebalance collapsing root (ITS#8062)
Fix mdb_load with large values (ITS#8066)
Fix to retry writes on EINTR (ITS#8106)
Fix mdb_cursor_del on empty DB (ITS#8109)
Fix and Rework comparison for MDB_INTEGERKEY/MDB_INTEGERDUP (ITS#8117)
Fix error handling (ITS#7959,#8157,etc.)
Fix race conditions (ITS#7969,7970)
Added workaround for fdatasync bug in ext3fs
Build
Don't use -fPIC for static lib
Update .gitignore (ITS#7952,#7953)
Cleanup for "make test" (ITS#7841), "make clean", mtest*.c
Misc. Android/Windows cleanup
Documentation
Fix MDB_APPEND doc
Fix MDB_MAXKEYSIZE doc (ITS#8156)
Fix mdb_cursor_put,mdb_cursor_del EACCES description
Fix mdb_env_sync(MDB_RDONLY env) doc (ITS#8021)
Clarify MDB_WRITEMAP doc (ITS#8021)
Clarify mdb_env_open doc
Clarify mdb_dbi_open doc
LMDB 0.9.14 Release (2014/09/20)
Fix to support 64K page size (ITS#7713)
Fix to persist decreased as well as increased mapsizes (ITS#7789)
Fix cursor bug when deleting last node of a DUPSORT key
Fix mdb_env_info to return FIXEDMAP address
Fix ambiguous error code from writing to closed DBI (ITS#7825)
Fix mdb_copy copying past end of file (ITS#7886)
Fix cursor bugs from page_merge/rebalance
Fix to dirty fewer pages in deletes (mdb_page_loose())
Fix mdb_dbi_open creating subDBs (ITS#7917)
Fix mdb_cursor_get(_DUP) with single value (ITS#7913)
Fix Windows compat issues in mtests (ITS#7879)
Add compacting variant of mdb_copy
Add BigEndian integer key compare code
Add mdb_dump/mdb_load utilities
LMDB 0.9.13 Release (2014/06/18)
Fix mdb_page_alloc unlimited overflow page search
Documentation
Re-fix MDB_CURRENT doc (ITS#7793)
Fix MDB_GET_MULTIPLE/MDB_NEXT_MULTIPLE doc
LMDB 0.9.12 Release (2014/06/13)
Fix MDB_GET_BOTH regression (ITS#7875,#7681)
Fix MDB_MULTIPLE writing multiple keys (ITS#7834)
Fix mdb_rebalance (ITS#7829)
Fix mdb_page_split (ITS#7815)
Fix md_entries count (ITS#7861,#7828,#7793)
Fix MDB_CURRENT (ITS#7793)
Fix possible crash on Windows DLL detach
Misc code cleanup
Documentation
mdb_cursor_put: cursor moves on error (ITS#7771)
LMDB 0.9.11 Release (2014/01/15)
Add mdb_env_set_assert() (ITS#7775)
Fix: invalidate txn on page allocation errors (ITS#7377)
Fix xcursor tracking in mdb_cursor_del0() (ITS#7771)
Fix corruption from deletes (ITS#7756)
Fix Windows/MSVC build issues
Raise safe limit of max MDB_MAXKEYSIZE
Misc code cleanup
Documentation
Remove spurious note about non-overlapping flags (ITS#7665)
LMDB 0.9.10 Release (2013/11/12)
Add MDB_NOMEMINIT option
Fix mdb_page_split() again (ITS#7589)
Fix MDB_NORDAHEAD definition (ITS#7734)
Fix mdb_cursor_del() positioning (ITS#7733)
Partial fix for larger page sizes (ITS#7713)
Fix Windows64/MSVC build issues
LMDB 0.9.9 Release (2013/10/24)
Add mdb_env_get_fd()
Add MDB_NORDAHEAD option
Add MDB_NOLOCK option
Avoid wasting space in mdb_page_split() (ITS#7589)
Fix mdb_page_merge() cursor fixup (ITS#7722)
Fix mdb_cursor_del() on last delete (ITS#7718)
Fix adding WRITEMAP on existing env (ITS#7715)
Fix nested txns (ITS#7515)
Fix mdb_env_copy() O_DIRECT bug (ITS#7682)
Fix mdb_cursor_set(SET_RANGE) return code (ITS#7681)
Fix mdb_rebalance() cursor fixup (ITS#7701)
Misc code cleanup
Documentation
Note that by default, readers need write access
LMDB 0.9.8 Release (2013/09/09)
Allow mdb_env_set_mapsize() on an open environment
Fix mdb_dbi_flags() (ITS#7672)
Fix mdb_page_unspill() in nested txns
Fix mdb_cursor_get(CURRENT|NEXT) after a delete
Fix mdb_cursor_get(DUP) to always return key (ITS#7671)
Fix mdb_cursor_del() to always advance to next item (ITS#7670)
Fix mdb_cursor_set(SET_RANGE) for tree with single page (ITS#7681)
Fix mdb_env_copy() retry open if O_DIRECT fails (ITS#7682)
Tweak mdb_page_spill() to be less aggressive
Documentation
Update caveats since mdb_reader_check() added in 0.9.7
LMDB 0.9.7 Release (2013/08/17)
Don't leave stale lockfile on failed RDONLY open (ITS#7664)
Fix mdb_page_split() ref beyond cursor depth
Fix read txn data race (ITS#7635)
Fix mdb_rebalance (ITS#7536, #7538)
Fix mdb_drop() (ITS#7561)
Misc DEBUG macro fixes
Add MDB_NOTLS envflag
Add mdb_env_copyfd()
Add mdb_txn_env() (ITS#7660)
Add mdb_dbi_flags() (ITS#7661)
Add mdb_env_get_maxkeysize()
Add mdb_env_reader_list()/mdb_env_reader_check()
Add mdb_page_spill/unspill, remove hard txn size limit
Use shorter names for semaphores (ITS#7615)
Build
Fix install target (ITS#7656)
Documentation
Misc updates for cursors, DB handles, data lifetime
LMDB 0.9.6 Release (2013/02/25)
Many fixes/enhancements
LMDB 0.9.5 Release (2012/11/30)
Renamed from libmdb to liblmdb
Many fixes/enhancements

1631
Doxyfile
View File

File diff suppressed because it is too large Load Diff

39
Makefile
View File

@ -36,24 +36,21 @@ IOARENA ?= ../ioarena.git/@BUILD/src/ioarena
########################################################################
HEADERS := lmdb.h mdbx.h
HEADERS := mdbx.h
LIBRARIES := libmdbx.a libmdbx.so
TOOLS := mdbx_stat mdbx_copy mdbx_dump mdbx_load mdbx_chk
MANPAGES := mdb_stat.1 mdb_copy.1 mdb_dump.1 mdb_load.1
MANPAGES := mdbx_stat.1 mdbx_copy.1 mdbx_dump.1 mdbx_load.1
TESTS := mtest0 mtest1 mtest2 mtest3 mtest4 mtest5 mtest6 wbench \
yota_test1 yota_test2
SRC_LMDB := mdb.c midl.c lmdb.h midl.h reopen.h barriers.h
SRC_MDBX := $(SRC_LMDB) mdbx.c mdbx.h
SRC_MDBX := mdbx.c mdbx.h reopen.h barriers.h
.PHONY: mdbx lmdb all install clean check tests coverage
.PHONY: mdbx all install clean check tests coverage
all: $(LIBRARIES) $(TOOLS)
mdbx: libmdbx.a libmdbx.so
lmdb: liblmdb.a liblmdb.so
tools: $(TOOLS)
install: $(LIBRARIES) $(TOOLS) $(HEADERS)
@ -88,25 +85,19 @@ libmdbx.a: mdbx.o
libmdbx.so: mdbx.lo
$(CC) $(CFLAGS) $(LDFLAGS) -save-temps -pthread -shared $(LDOPS) -o $@ $^
liblmdb.a: lmdb.o
$(AR) rs $@ $^
liblmdb.so: lmdb.lo
$(CC) $(CFLAGS) $(LDFLAGS) -pthread -shared $(LDOPS) -o $@ $^
mdbx_stat: mdb_stat.o mdbx.o
mdbx_stat: mdbx_stat.o mdbx.o
$(CC) $(CFLAGS) $(LDFLAGS) $(LDOPS) -o $@ $^
mdbx_copy: mdb_copy.o mdbx.o
mdbx_copy: mdbx_copy.o mdbx.o
$(CC) $(CFLAGS) $(LDFLAGS) $(LDOPS) -o $@ $^
mdbx_dump: mdb_dump.o mdbx.o
mdbx_dump: mdbx_dump.o mdbx.o
$(CC) $(CFLAGS) $(LDFLAGS) $(LDOPS) -o $@ $^
mdbx_load: mdb_load.o mdbx.o
mdbx_load: mdbx_load.o mdbx.o
$(CC) $(CFLAGS) $(LDFLAGS) $(LDOPS) -o $@ $^
mdbx_chk: mdb_chk.o mdbx.o
mdbx_chk: mdbx_chk.o mdbx.o
$(CC) $(CFLAGS) $(LDFLAGS) $(LDOPS) -o $@ $^
mtest0: mtest0.o mdbx.o
@ -145,16 +136,10 @@ mdbx.o: $(SRC_MDBX)
mdbx.lo: $(SRC_MDBX)
$(CC) $(CFLAGS) -fPIC -c mdbx.c -o $@
lmdb.o: $(SRC_LMDB)
$(CC) $(CFLAGS) -c mdb.c -o $@
lmdb.lo: $(SRC_LMDB)
$(CC) $(CFLAGS) -fPIC -c mdb.c -o $@
%: %.o
$(CC) $(CFLAGS) $(LDFLAGS) $^ -o $@
%.o: %.c lmdb.h mdbx.h
%.o: %.c mdbx.h
$(CC) $(CFLAGS) -c $<
COFLAGS = -fprofile-arcs -ftest-coverage
@ -188,13 +173,11 @@ endef
$(eval $(call bench-rule,mdbx,$(NN),libmdbx.so))
$(eval $(call bench-rule,lmdb,$(NN)))
$(eval $(call bench-rule,dummy,$(NN)))
$(eval $(call bench-rule,debug,10))
bench: bench-lmdb.txt bench-mdbx.txt
bench: bench-mdbx.txt
endif

6
README.md
View File

@ -194,7 +194,7 @@ RECLAIM` в _libmdbx_.
сохранены мета-страницы со ссылками на страницы с новыми
версиями данных, но не сами новые данные. В этом случае БД
будет безвозвратна разрушена, даже если до аварии производилась
полная синхронизация данных (посредством `mdb_env_sync()`).
полная синхронизация данных (посредством `mdbx_env_sync()`).
В _libmdbx_ эта проблема устранена, подробности ниже.
@ -248,7 +248,7 @@ RECLAIM` в _libmdbx_.
сохранены мета-страницы со ссылками на страницы с новыми
версиями данных, но не сами новые данные. В этом случае БД
будет безвозвратна разрушена, даже если до аварии производилась
полная синхронизация данных (посредством `mdb_env_sync()`).
полная синхронизация данных (посредством `mdbx_env_sync()`).
В _libmdbx_ эта проблема устранена путем полной переработки
пути записи данных:
@ -371,5 +371,5 @@ RECLAIM` в _libmdbx_.
25. При завершении читающих транзакций, открытые в них DBI-хендлы не
закрываются и не теряются при завершении таких транзакций посредством
mdb_txn_abort() или mdb_txn_reset(). Что позволяет избавится от ряда
mdbx_txn_abort() или mdbx_txn_reset(). Что позволяет избавится от ряда
сложно обнаруживаемых ошибок.

2
barriers.h
View File

@ -140,7 +140,7 @@ static MDBX_INLINE void mdbx_barrier(int type) {
#define mdbx_coherent_barrier() \
mdbx_barrier(MDB_CACHE_IS_COHERENT ? MDBX_BARRIER_COMPILER : MDBX_BARRIER_MEMORY)
static MDBX_INLINE void mdb_invalidate_cache(void *addr, int nbytes) {
static MDBX_INLINE void mdbx_invalidate_cache(void *addr, int nbytes) {
mdbx_coherent_barrier();
#if defined(__mips) && defined(__linux)
/* MIPS has cache coherency issues.

1557
lmdb.h
View File

File diff suppressed because it is too large Load Diff

10723
mdb.c
View File

File diff suppressed because it is too large Load Diff

954
mdb_chk.c
View File

@ -1,954 +0,0 @@
/* mdbx_chk.c - memory-mapped database check tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2015,2016 Peter-Service R&D LLC.
*
* This file is part of libmdbx.
*
* libmdbx is free software; you can redistribute it and/or modify it under
* the terms of the GNU Affero General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* (at your option) any later version.
*
* libmdbx is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
*/
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <unistd.h>
#include <signal.h>
#include <stdarg.h>
#include <malloc.h>
#include <time.h>
#include "midl.h"
#include "mdbx.h"
typedef struct flagbit {
int bit;
char *name;
} flagbit;
flagbit dbflags[] = {
{ MDB_DUPSORT, "dupsort" },
{ MDB_INTEGERKEY, "integerkey" },
{ MDB_REVERSEKEY, "reversekey" },
{ MDB_DUPFIXED, "dupfixed" },
{ MDB_REVERSEDUP, "reversedup" },
{ MDB_INTEGERDUP, "integerdup" },
{ 0, NULL }
};
static volatile sig_atomic_t gotsignal;
static void signal_handler( int sig ) {
(void) sig;
gotsignal = 1;
}
#define MAX_DBI 32768
#define EXIT_INTERRUPTED (EXIT_FAILURE+4)
#define EXIT_FAILURE_SYS (EXIT_FAILURE+3)
#define EXIT_FAILURE_MDB (EXIT_FAILURE+2)
#define EXIT_FAILURE_CHECK_MAJOR (EXIT_FAILURE+1)
#define EXIT_FAILURE_CHECK_MINOR EXIT_FAILURE
struct {
const char* dbi_names[MAX_DBI];
size_t dbi_pages[MAX_DBI];
size_t dbi_empty_pages[MAX_DBI];
size_t dbi_payload_bytes[MAX_DBI];
size_t dbi_lost_bytes[MAX_DBI];
short *pagemap;
size_t total_payload_bytes;
size_t pgcount;
} walk;
static __attribute__((constructor))
void init_walk(void)
{
walk.dbi_names[0] = "@gc";
}
size_t total_unused_bytes;
int exclusive = 2;
MDB_env *env;
MDB_txn *txn, *locktxn;
MDBX_envinfo info;
MDBX_stat stat;
size_t maxkeysize, reclaimable_pages, freedb_pages, lastpgno;
size_t userdb_count, skipped_subdb;
unsigned verbose, quiet;
const char* only_subdb;
struct problem {
struct problem* pr_next;
size_t count;
const char* caption;
};
struct problem* problems_list;
size_t total_problems;
static void __attribute__ ((format (printf, 1, 2)))
print(const char* msg, ...) {
if (! quiet) {
va_list args;
fflush(stderr);
va_start(args, msg);
vfprintf(stdout, msg, args);
va_end(args);
}
}
static void __attribute__ ((format (printf, 1, 2)))
error(const char* msg, ...) {
total_problems++;
if (! quiet) {
va_list args;
fflush(stdout);
va_start(args, msg);
vfprintf(stderr, msg, args);
va_end(args);
fflush(NULL);
}
}
static void pagemap_cleanup(void) {
int i;
for( i = 1; i < MAX_DBI; ++i ) {
if (walk.dbi_names[i]) {
free((void *) walk.dbi_names[i]);
walk.dbi_names[i] = NULL;
}
}
free(walk.pagemap);
walk.pagemap = NULL;
}
static int pagemap_lookup_dbi(const char* dbi) {
static int last;
int i;
if (last > 0 && strcmp(walk.dbi_names[last], dbi) == 0)
return last;
for(i = 1; walk.dbi_names[i] && last < MAX_DBI; ++i)
if (strcmp(walk.dbi_names[i], dbi) == 0)
return last = i;
if (i == MAX_DBI)
return -1;
walk.dbi_names[i] = strdup(dbi);
if (verbose > 1) {
print(" - found '%s' area\n", dbi);
fflush(NULL);
}
return last = i;
}
static void problem_add(const char* object, size_t entry_number, const char* msg, const char *extra, ...) {
total_problems++;
if (! quiet) {
int need_fflush = 0;
struct problem* p;
for (p = problems_list; p; p = p->pr_next)
if (p->caption == msg)
break;
if (! p) {
p = calloc(1, sizeof(*p));
p->caption = msg;
p->pr_next = problems_list;
problems_list = p;
need_fflush = 1;
}
p->count++;
if (verbose > 1) {
print(" %s #%zu: %s", object, entry_number, msg);
if (extra) {
va_list args;
printf(" (");
va_start(args, extra);
vfprintf(stdout, extra, args);
va_end(args);
printf(")");
}
printf("\n");
if (need_fflush)
fflush(NULL);
}
}
}
static struct problem* problems_push() {
struct problem* p = problems_list;
problems_list = NULL;
return p;
}
static size_t problems_pop(struct problem* list) {
size_t count = 0;
if (problems_list) {
int i;
print(" - problems: ");
for (i = 0; problems_list; ++i) {
struct problem* p = problems_list->pr_next;
count += problems_list->count;
print("%s%s (%zu)", i ? ", " : "", problems_list->caption, problems_list->count);
free(problems_list);
problems_list = p;
}
print("\n");
fflush(NULL);
}
problems_list = list;
return count;
}
static int pgvisitor(size_t pgno, unsigned pgnumber, void* ctx, const char* dbi,
const char* type, int nentries, int payload_bytes, int header_bytes, int unused_bytes)
{
(void) ctx;
if (type) {
size_t page_bytes = payload_bytes + header_bytes + unused_bytes;
size_t page_size = pgnumber * stat.base.ms_psize;
int index = pagemap_lookup_dbi(dbi);
if (index < 0)
return ENOMEM;
if (verbose > 2 && (!only_subdb || strcmp(only_subdb, dbi) == 0)) {
if (pgnumber == 1)
print(" %s-page %zu", type, pgno);
else
print(" %s-span %zu[%u]", type, pgno, pgnumber);
print(" of %s: header %i, payload %i, unused %i\n",
dbi, header_bytes, payload_bytes, unused_bytes);
}
walk.pgcount += pgnumber;
if (unused_bytes < 0 || (size_t) unused_bytes > page_size)
problem_add("page", pgno, "illegal unused-bytes", "%zu < %i < %zu",
0, unused_bytes, stat.base.ms_psize);
if (header_bytes < (int) sizeof(long) || (size_t) header_bytes >= stat.base.ms_psize - sizeof(long))
problem_add("page", pgno, "illegal header-length", "%zu < %i < %zu",
sizeof(long), header_bytes, stat.base.ms_psize - sizeof(long));
if (payload_bytes < 1) {
if (nentries > 1) {
problem_add("page", pgno, "zero size-of-entry", "payload %i bytes, %i entries",
payload_bytes, nentries);
if ((size_t) header_bytes + unused_bytes < page_size) {
/* LY: hush a misuse error */
page_bytes = page_size;
}
} else {
problem_add("page", pgno, "empty", "payload %i bytes, %i entries",
payload_bytes, nentries);
walk.dbi_empty_pages[index] += 1;
}
}
if (page_bytes != page_size) {
problem_add("page", pgno, "misused", "%zu != %zu (%ih + %ip + %iu)",
page_size, page_bytes, header_bytes, payload_bytes, unused_bytes);
if (page_size > page_bytes)
walk.dbi_lost_bytes[index] += page_size - page_bytes;
} else {
walk.dbi_payload_bytes[index] += payload_bytes + header_bytes;
walk.total_payload_bytes += payload_bytes + header_bytes;
}
if (pgnumber) {
do {
if (pgno >= lastpgno)
problem_add("page", pgno, "wrong page-no",
"%zu > %zi", pgno, lastpgno);
else if (walk.pagemap[pgno])
problem_add("page", pgno, "already used",
"in %s", walk.dbi_names[walk.pagemap[pgno]]);
else {
walk.pagemap[pgno] = index;
walk.dbi_pages[index] += 1;
}
++pgno;
} while(--pgnumber);
}
}
return gotsignal ? EINTR : MDB_SUCCESS;
}
typedef int (visitor)(size_t record_number, MDB_val *key, MDB_val* data);
static int process_db(MDB_dbi dbi, char *name, visitor *handler, int silent);
static int handle_userdb(size_t record_number, MDB_val *key, MDB_val* data) {
(void) record_number;
(void) key;
(void) data;
return MDB_SUCCESS;
}
static int handle_freedb(size_t record_number, MDB_val *key, MDB_val* data) {
char *bad = "";
size_t pg, prev;
ssize_t i, number, span = 0;
size_t *iptr = data->mv_data, txnid = *(size_t*)key->mv_data;
if (key->mv_size != sizeof(txnid))
problem_add("entry", record_number, "wrong txn-id size", "key-size %zi", key->mv_size);
else if (txnid < 1 || txnid > info.base.me_last_txnid)
problem_add("entry", record_number, "wrong txn-id", "%zu", txnid);
if (data->mv_size < sizeof(size_t) || data->mv_size % sizeof(size_t))
problem_add("entry", record_number, "wrong idl size", "%zu", data->mv_size);
else {
number = *iptr++;
if (number >= MDB_IDL_UM_MAX)
problem_add("entry", record_number, "wrong idl length", "%zi", number);
else if ((number + 1) * sizeof(size_t) != data->mv_size)
problem_add("entry", record_number, "mismatch idl length", "%zi != %zu",
number * sizeof(size_t), data->mv_size);
else {
freedb_pages += number;
if (info.me_tail_txnid > txnid)
reclaimable_pages += number;
for (i = number, prev = 1; --i >= 0; ) {
pg = iptr[i];
if (pg < 2 /* META_PAGE */ || pg > info.base.me_last_pgno)
problem_add("entry", record_number, "wrong idl entry", "2 < %zi < %zi",
pg, info.base.me_last_pgno);
else if (pg <= prev) {
bad = " [bad sequence]";
problem_add("entry", record_number, "bad sequence", "%zi <= %zi",
pg, prev);
}
prev = pg;
pg += span;
for (; i >= span && iptr[i - span] == pg; span++, pg++) ;
}
if (verbose > 2 && !only_subdb) {
print(" transaction %zu, %zd pages, maxspan %zd%s\n",
*(size_t *)key->mv_data, number, span, bad);
if (verbose > 3) {
int j = number - 1;
while (j >= 0) {
pg = iptr[j];
for (span = 1; --j >= 0 && iptr[j] == pg + span; span++) ;
if (span > 1)
print(" %9zu[%zd]\n", pg, span);
else
print(" %9zu\n", pg);
}
}
}
}
}
return MDB_SUCCESS;
}
static int handle_maindb(size_t record_number, MDB_val *key, MDB_val* data) {
char *name;
int rc;
size_t i;
name = key->mv_data;
for(i = 0; i < key->mv_size; ++i) {
if (name[i] < ' ')
return handle_userdb(record_number, key, data);
}
name = malloc(key->mv_size + 1);
memcpy(name, key->mv_data, key->mv_size);
name[key->mv_size] = '\0';
userdb_count++;
rc = process_db(-1, name, handle_userdb, 0);
free(name);
if (rc != MDB_INCOMPATIBLE)
return rc;
return handle_userdb(record_number, key, data);
}
static int process_db(MDB_dbi dbi, char *name, visitor *handler, int silent)
{
MDB_cursor *mc;
MDBX_stat ms;
MDB_val key, data;
MDB_val prev_key, prev_data;
unsigned flags;
int rc, i;
struct problem* saved_list;
size_t problems_count;
unsigned record_count = 0, dups = 0;
size_t key_bytes = 0, data_bytes = 0;
if (0 > (int) dbi) {
rc = mdbx_dbi_open(txn, name, 0, &dbi);
if (rc) {
if (!name || rc != MDB_INCOMPATIBLE) /* LY: mainDB's record is not a user's DB. */ {
error(" - mdbx_open '%s' failed, error %d %s\n",
name ? name : "main", rc, mdbx_strerror(rc));
}
return rc;
}
}
if (dbi >= 2 /* CORE_DBS */ && name && only_subdb && strcmp(only_subdb, name)) {
if (verbose) {
print("Skip processing '%s'...\n", name);
fflush(NULL);
}
skipped_subdb++;
return MDB_SUCCESS;
}
if (! silent && verbose) {
print("Processing '%s'...\n", name ? name : "main");
fflush(NULL);
}
rc = mdbx_dbi_flags(txn, dbi, &flags);
if (rc) {
error(" - mdbx_dbi_flags failed, error %d %s\n", rc, mdbx_strerror(rc));
return rc;
}
rc = mdbx_stat(txn, dbi, &ms, sizeof(ms));
if (rc) {
error(" - mdbx_stat failed, error %d %s\n", rc, mdbx_strerror(rc));
return rc;
}
if (! silent && verbose) {
print(" - dbi-id %d, flags:", dbi);
if (! flags)
print(" none");
else {
for (i=0; dbflags[i].bit; i++)
if (flags & dbflags[i].bit)
print(" %s", dbflags[i].name);
}
print(" (0x%02X)\n", flags);
if (verbose > 1) {
print(" - page size %u, entries %zu\n", ms.base.ms_psize, ms.base.ms_entries);
print(" - b-tree depth %u, pages: branch %zu, leaf %zu, overflow %zu\n",
ms.base.ms_depth, ms.base.ms_branch_pages, ms.base.ms_leaf_pages, ms.base.ms_overflow_pages);
}
}
rc = mdbx_cursor_open(txn, dbi, &mc);
if (rc) {
error(" - mdbx_cursor_open failed, error %d %s\n", rc, mdbx_strerror(rc));
return rc;
}
saved_list = problems_push();
prev_key.mv_data = NULL;
prev_data.mv_size = 0;
rc = mdbx_cursor_get(mc, &key, &data, MDB_FIRST);
while (rc == MDB_SUCCESS) {
if (gotsignal) {
print(" - interrupted by signal\n");
fflush(NULL);
rc = EINTR;
goto bailout;
}
if (key.mv_size > maxkeysize) {
problem_add("entry", record_count, "key length exceeds max-key-size",
"%zu > %zu", key.mv_size, maxkeysize);
} else if ((flags & MDB_INTEGERKEY)
&& key.mv_size != sizeof(size_t) && key.mv_size != sizeof(int)) {
problem_add("entry", record_count, "wrong key length",
"%zu != %zu", key.mv_size, sizeof(size_t));
}
if ((flags & MDB_INTEGERDUP)
&& data.mv_size != sizeof(size_t) && data.mv_size != sizeof(int)) {
problem_add("entry", record_count, "wrong data length",
"%zu != %zu", data.mv_size, sizeof(size_t));
}
if (prev_key.mv_data) {
if ((flags & MDB_DUPFIXED) && prev_data.mv_size != data.mv_size) {
problem_add("entry", record_count, "different data length",
"%zu != %zu", prev_data.mv_size, data.mv_size);
}
int cmp = mdbx_cmp(txn, dbi, &prev_key, &key);
if (cmp > 0) {
problem_add("entry", record_count, "broken ordering of entries", NULL);
} else if (cmp == 0) {
++dups;
if (! (flags & MDB_DUPSORT))
problem_add("entry", record_count, "duplicated entries", NULL);
else if (flags & MDB_INTEGERDUP) {
cmp = mdbx_dcmp(txn, dbi, &prev_data, &data);
if (cmp > 0)
problem_add("entry", record_count, "broken ordering of multi-values", NULL);
}
}
} else if (verbose) {
if (flags & MDB_INTEGERKEY)
print(" - fixed key-size %zu\n", key.mv_size );
if (flags & (MDB_INTEGERDUP | MDB_DUPFIXED))
print(" - fixed data-size %zu\n", data.mv_size );
}
if (handler) {
rc = handler(record_count, &key, &data);
if (rc)
goto bailout;
}
record_count++;
key_bytes += key.mv_size;
data_bytes += data.mv_size;
prev_key = key;
prev_data = data;
rc = mdbx_cursor_get(mc, &key, &data, MDB_NEXT);
}
if (rc != MDB_NOTFOUND)
error(" - mdbx_cursor_get failed, error %d %s\n", rc, mdbx_strerror(rc));
else
rc = 0;
if (record_count != ms.base.ms_entries)
problem_add("entry", record_count, "differentent number of entries",
"%zu != %zu", record_count, ms.base.ms_entries);
bailout:
problems_count = problems_pop(saved_list);
if (! silent && verbose) {
print(" - summary: %u records, %u dups, %zu key's bytes, %zu data's bytes, %zu problems\n",
record_count, dups, key_bytes, data_bytes, problems_count);
fflush(NULL);
}
mdbx_cursor_close(mc);
return rc || problems_count;
}
static void usage(char *prog)
{
fprintf(stderr, "usage: %s dbpath [-V] [-v] [-n] [-q] [-w] [-c] [-d] [-s subdb]\n"
" -V\t\tshow version\n"
" -v\t\tmore verbose, could be used multiple times\n"
" -n\t\tNOSUBDIR mode for open\n"
" -q\t\tbe quiet\n"
" -w\t\tlock DB for writing while checking\n"
" -d\t\tdisable page-by-page traversal of b-tree\n"
" -s subdb\tprocess a specific subdatabase only\n"
" -c\t\tforce cooperative mode (don't try exclusive)\n", prog);
exit(EXIT_INTERRUPTED);
}
const char* meta_synctype(size_t sign) {
switch(sign) {
case 0:
return "no-sync/legacy";
case 1:
return "weak";
default:
return "steady";
}
}
int meta_lt(size_t txn1, size_t sign1, size_t txn2, size_t sign2) {
return ((sign1 > 1) == (sign2 > 1)) ? txn1 < txn2 : txn2 && sign2 > 1;
}
int main(int argc, char *argv[])
{
int i, rc;
char *prog = argv[0];
char *envname;
int envflags = MDB_RDONLY;
int problems_maindb = 0, problems_freedb = 0, problems_meta = 0;
int dont_traversal = 0;
size_t n;
struct timespec timestamp_start, timestamp_finish;
double elapsed;
atexit(pagemap_cleanup);
if (clock_gettime(CLOCK_MONOTONIC, &timestamp_start)) {
rc = errno;
error("clock_gettime failed, error %d %s\n", rc, mdbx_strerror(rc));
return EXIT_FAILURE_SYS;
}
if (argc < 2) {
usage(prog);
}
while ((i = getopt(argc, argv, "Vvqnwcds:")) != EOF) {
switch(i) {
case 'V':
printf("%s\n", MDB_VERSION_STRING);
exit(EXIT_SUCCESS);
break;
case 'v':
verbose++;
break;
case 'q':
quiet = 1;
break;
case 'n':
envflags |= MDB_NOSUBDIR;
break;
case 'w':
envflags &= ~MDB_RDONLY;
break;
case 'c':
exclusive = 0;
break;
case 'd':
dont_traversal = 1;
break;
case 's':
if (only_subdb && strcmp(only_subdb, optarg))
usage(prog);
only_subdb = optarg;
break;
default:
usage(prog);
}
}
if (optind != argc - 1)
usage(prog);
#ifdef SIGPIPE
signal(SIGPIPE, signal_handler);
#endif
#ifdef SIGHUP
signal(SIGHUP, signal_handler);
#endif
signal(SIGINT, signal_handler);
signal(SIGTERM, signal_handler);
envname = argv[optind];
print("Running mdbx_chk for '%s' in %s mode...\n",
envname, (envflags & MDB_RDONLY) ? "read-only" : "write-lock");
fflush(NULL);
rc = mdbx_env_create(&env);
if (rc) {
error("mdbx_env_create failed, error %d %s\n", rc, mdbx_strerror(rc));
return rc < 0 ? EXIT_FAILURE_MDB : EXIT_FAILURE_SYS;
}
rc = mdbx_env_get_maxkeysize(env);
if (rc < 0) {
error("mdbx_env_get_maxkeysize failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
maxkeysize = rc;
rc = mdbx_env_set_maxdbs(env, MAX_DBI);
if (rc < 0) {
error("mdbx_env_set_maxdbs failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
rc = mdbx_env_open_ex(env, envname, envflags, 0664, &exclusive);
if (rc) {
error("mdbx_env_open failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
if (verbose)
print(" - %s mode\n", exclusive ? "monopolistic" : "cooperative");
if (! (envflags & MDB_RDONLY)) {
rc = mdbx_txn_begin(env, NULL, 0, &locktxn);
if (rc) {
error("mdbx_txn_begin(lock-write) failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
}
rc = mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn);
if (rc) {
error("mdbx_txn_begin(read-only) failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
rc = mdbx_env_info(env, &info, sizeof(info));
if (rc) {
error("mdbx_env_info failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
rc = mdbx_env_stat(env, &stat, sizeof(stat));
if (rc) {
error("mdbx_env_stat failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
lastpgno = info.base.me_last_pgno + 1;
errno = 0;
if (verbose) {
double k = 1024.0;
const char sf[] = "KMGTPEZY"; /* LY: Kilo, Mega, Giga, Tera, Peta, Exa, Zetta, Yotta! */
for(i = 0; sf[i+1] && info.base.me_mapsize / k > 1000.0; ++i)
k *= 1024;
print(" - map size %zu (%.2f %cb)\n", info.base.me_mapsize,
info.base.me_mapsize / k, sf[i]);
if (info.base.me_mapaddr)
print(" - mapaddr %p\n", info.base.me_mapaddr);
print(" - pagesize %u, max keysize %zu (%s), max readers %u\n",
stat.base.ms_psize, maxkeysize,
(maxkeysize == 511) ? "default" :
(maxkeysize == 0) ? "devel" : "custom",
info.base.me_maxreaders);
print(" - transactions: last %zu, bottom %zu, lag reading %zi\n", info.base.me_last_txnid,
info.me_tail_txnid, info.base.me_last_txnid - info.me_tail_txnid);
print(" - meta-1: %s %zu, %s",
meta_synctype(info.me_meta1_sign), info.me_meta1_txnid,
meta_lt(info.me_meta1_txnid, info.me_meta1_sign,
info.me_meta2_txnid, info.me_meta2_sign) ? "tail" : "head");
if (info.me_meta1_txnid > info.base.me_last_txnid)
print(", rolled-back %zu (%zu >>> %zu)",
info.me_meta1_txnid - info.base.me_last_txnid,
info.me_meta1_txnid, info.base.me_last_txnid);
print("\n");
print(" - meta-2: %s %zu, %s",
meta_synctype(info.me_meta2_sign), info.me_meta2_txnid,
meta_lt(info.me_meta2_txnid, info.me_meta2_sign,
info.me_meta1_txnid, info.me_meta1_sign) ? "tail" : "head");
if (info.me_meta2_txnid > info.base.me_last_txnid)
print(", rolled-back %zu (%zu >>> %zu)",
info.me_meta2_txnid - info.base.me_last_txnid,
info.me_meta2_txnid, info.base.me_last_txnid);
print("\n");
}
if (exclusive > 1) {
if (verbose)
print(" - perform full check last-txn-id with meta-pages\n");
if (! meta_lt(info.me_meta1_txnid, info.me_meta1_sign,
info.me_meta2_txnid, info.me_meta2_sign)
&& info.me_meta1_txnid != info.base.me_last_txnid) {
print(" - meta-1 txn-id mismatch last-txn-id (%zi != %zi)\n",
info.me_meta1_txnid, info.base.me_last_txnid);
++problems_meta;
}
if (! meta_lt(info.me_meta2_txnid, info.me_meta2_sign,
info.me_meta1_txnid, info.me_meta1_sign)
&& info.me_meta2_txnid != info.base.me_last_txnid) {
print(" - meta-2 txn-id mismatch last-txn-id (%zi != %zi)\n",
info.me_meta2_txnid, info.base.me_last_txnid);
++problems_meta;
}
} else if (locktxn) {
if (verbose)
print(" - perform lite check last-txn-id with meta-pages (not a monopolistic mode)\n");
size_t last = (info.me_meta2_txnid > info.me_meta1_txnid) ? info.me_meta2_txnid : info.me_meta1_txnid;
if (last != info.base.me_last_txnid) {
print(" - last-meta mismatch last-txn-id (%zi != %zi)\n",
last, info.base.me_last_txnid);
++problems_meta;
}
} else if (verbose) {
print(" - skip check last-txn-id with meta-pages (monopolistic or write-lock mode only)\n");
}
if (!dont_traversal) {
struct problem* saved_list;
size_t traversal_problems;
size_t empty_pages, lost_bytes;
print("Traversal b-tree...\n");
fflush(NULL);
walk.pagemap = calloc(lastpgno, sizeof(*walk.pagemap));
if (! walk.pagemap) {
rc = errno ? errno : ENOMEM;
error("calloc failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
saved_list = problems_push();
rc = mdbx_env_pgwalk(txn, pgvisitor, NULL);
traversal_problems = problems_pop(saved_list);
if (rc) {
if (rc == EINTR && gotsignal) {
print(" - interrupted by signal\n");
fflush(NULL);
} else {
error("mdbx_env_pgwalk failed, error %d %s\n", rc, mdbx_strerror(rc));
}
goto bailout;
}
for( n = 0; n < lastpgno; ++n)
if (! walk.pagemap[n])
walk.dbi_pages[0] += 1;
empty_pages = lost_bytes = 0;
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
empty_pages += walk.dbi_empty_pages[i];
lost_bytes += walk.dbi_lost_bytes[i];
}
if (verbose) {
size_t total_page_bytes = walk.pgcount * stat.base.ms_psize;
print(" - dbi pages: %zu total", walk.pgcount);
if (verbose > 1)
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i)
print(", %s %zu", walk.dbi_names[i], walk.dbi_pages[i]);
print(", %s %zu\n", walk.dbi_names[0], walk.dbi_pages[0]);
if (verbose > 1) {
print(" - space info: total %zu bytes, payload %zu (%.1f%%), unused %zu (%.1f%%)\n",
total_page_bytes, walk.total_payload_bytes,
walk.total_payload_bytes * 100.0 / total_page_bytes,
total_page_bytes - walk.total_payload_bytes,
(total_page_bytes - walk.total_payload_bytes) * 100.0 / total_page_bytes);
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
size_t dbi_bytes = walk.dbi_pages[i] * stat.base.ms_psize;
print(" %s: subtotal %zu bytes (%.1f%%), payload %zu (%.1f%%), unused %zu (%.1f%%)",
walk.dbi_names[i],
dbi_bytes, dbi_bytes * 100.0 / total_page_bytes,
walk.dbi_payload_bytes[i], walk.dbi_payload_bytes[i] * 100.0 / dbi_bytes,
dbi_bytes - walk.dbi_payload_bytes[i],
(dbi_bytes - walk.dbi_payload_bytes[i]) * 100.0 / dbi_bytes);
if (walk.dbi_empty_pages[i])
print(", %zu empty pages", walk.dbi_empty_pages[i]);
if (walk.dbi_lost_bytes[i])
print(", %zu bytes lost", walk.dbi_lost_bytes[i]);
print("\n");
}
}
print(" - summary: average fill %.1f%%", walk.total_payload_bytes * 100.0 / total_page_bytes);
if (empty_pages)
print(", %zu empty pages", empty_pages);
if (lost_bytes)
print(", %zu bytes lost", lost_bytes);
print(", %zu problems\n", traversal_problems);
}
} else if (verbose) {
print("Skipping b-tree walk...\n");
fflush(NULL);
}
if (! verbose)
print("Iterating DBIs...\n");
problems_maindb = process_db(-1, /* MAIN_DBI */ NULL, NULL, 0);
problems_freedb = process_db(0 /* FREE_DBI */, "free", handle_freedb, 0);
if (verbose) {
size_t value = info.base.me_mapsize / stat.base.ms_psize;
double percent = value / 100.0;
print(" - pages info: %zu total", value);
print(", allocated %zu (%.1f%%)", lastpgno, lastpgno / percent);
if (verbose > 1) {
value = info.base.me_mapsize / stat.base.ms_psize - lastpgno;
print(", remained %zu (%.1f%%)", value, value / percent);
value = lastpgno - freedb_pages;
print(", used %zu (%.1f%%)", value, value / percent);
print(", gc %zu (%.1f%%)", freedb_pages, freedb_pages / percent);
value = freedb_pages - reclaimable_pages;
print(", detained %zu (%.1f%%)", value, value / percent);
print(", reclaimable %zu (%.1f%%)", reclaimable_pages, reclaimable_pages / percent);
}
value = info.base.me_mapsize / stat.base.ms_psize - lastpgno + reclaimable_pages;
print(", available %zu (%.1f%%)\n", value, value / percent);
}
if (problems_maindb == 0 && problems_freedb == 0) {
if (!dont_traversal && (exclusive || locktxn)) {
if (walk.pgcount != lastpgno - freedb_pages) {
error("used pages mismatch (%zu != %zu)\n", walk.pgcount, lastpgno - freedb_pages);
}
if (walk.dbi_pages[0] != freedb_pages) {
error("gc pages mismatch (%zu != %zu)\n", walk.dbi_pages[0], freedb_pages);
}
} else if (verbose) {
print(" - skip check used and gc pages (btree-traversal with monopolistic or write-lock mode only)\n");
}
if (! process_db(-1, NULL, handle_maindb, 1)) {
if (! userdb_count && verbose)
print(" - does not contain multiple databases\n");
}
}
bailout:
if (txn)
mdbx_txn_abort(txn);
if (locktxn)
mdbx_txn_abort(locktxn);
if (env)
mdbx_env_close(env);
fflush(NULL);
if (rc) {
if (rc < 0)
return gotsignal ? EXIT_INTERRUPTED : EXIT_FAILURE_SYS;
return EXIT_FAILURE_MDB;
}
if (clock_gettime(CLOCK_MONOTONIC, &timestamp_finish)) {
rc = errno;
error("clock_gettime failed, error %d %s\n", rc, mdbx_strerror(rc));
return EXIT_FAILURE_SYS;
}
elapsed = timestamp_finish.tv_sec - timestamp_start.tv_sec
+ (timestamp_finish.tv_nsec - timestamp_start.tv_nsec) * 1e-9;
total_problems += problems_meta;
if (total_problems || problems_maindb || problems_freedb) {
print("Total %zu error(s) is detected, elapsed %.3f seconds.\n",
total_problems, elapsed);
if (problems_meta || problems_maindb || problems_freedb)
return EXIT_FAILURE_CHECK_MAJOR;
return EXIT_FAILURE_CHECK_MINOR;
}
print("No error is detected, elapsed %.3f seconds\n", elapsed);
return EXIT_SUCCESS;
}

81
mdb_copy.c
View File

@ -1,81 +0,0 @@
/* mdb_copy.c - memory-mapped database backup tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2012-2017 Howard Chu, Symas Corp.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include "mdbx.h"
static void
sighandle(int sig)
{
(void) sig;
}
int main(int argc,char * argv[])
{
int rc;
MDB_env *env = NULL;
const char *progname = argv[0], *act;
unsigned flags = MDB_RDONLY;
unsigned cpflags = 0;
for (; argc > 1 && argv[1][0] == '-'; argc--, argv++) {
if (argv[1][1] == 'n' && argv[1][2] == '\0')
flags |= MDB_NOSUBDIR;
else if (argv[1][1] == 'c' && argv[1][2] == '\0')
cpflags |= MDB_CP_COMPACT;
else if (argv[1][1] == 'V' && argv[1][2] == '\0') {
printf("%s\n", MDB_VERSION_STRING);
exit(0);
} else
argc = 0;
}
if (argc<2 || argc>3) {
fprintf(stderr, "usage: %s [-V] [-c] [-n] srcpath [dstpath]\n", progname);
exit(EXIT_FAILURE);
}
#ifdef SIGPIPE
signal(SIGPIPE, sighandle);
#endif
#ifdef SIGHUP
signal(SIGHUP, sighandle);
#endif
signal(SIGINT, sighandle);
signal(SIGTERM, sighandle);
act = "opening environment";
rc = mdb_env_create(&env);
if (rc == MDB_SUCCESS) {
rc = mdb_env_open(env, argv[1], flags, 0640);
}
if (rc == MDB_SUCCESS) {
act = "copying";
if (argc == 2)
rc = mdb_env_copyfd2(env, STDOUT_FILENO, cpflags);
else
rc = mdb_env_copy2(env, argv[2], cpflags);
}
if (rc)
fprintf(stderr, "%s: %s failed, error %d (%s)\n",
progname, act, rc, mdb_strerror(rc));
mdb_env_close(env);
return rc ? EXIT_FAILURE : EXIT_SUCCESS;
}

314
mdb_dump.c
View File

@ -1,314 +0,0 @@
/* mdb_dump.c - memory-mapped database dump tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2011-2017 Howard Chu, Symas Corp.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <unistd.h>
#include <signal.h>
#include "mdbx.h"
#define PRINT 1
static int mode;
typedef struct flagbit {
int bit;
char *name;
} flagbit;
flagbit dbflags[] = {
{ MDB_REVERSEKEY, "reversekey" },
{ MDB_DUPSORT, "dupsort" },
{ MDB_INTEGERKEY, "integerkey" },
{ MDB_DUPFIXED, "dupfixed" },
{ MDB_INTEGERDUP, "integerdup" },
{ MDB_REVERSEDUP, "reversedup" },
{ 0, NULL }
};
static volatile sig_atomic_t gotsig;
static void dumpsig( int sig )
{
(void) sig;
gotsig = 1;
}
static const char hexc[] = "0123456789abcdef";
static void hex(unsigned char c)
{
putchar(hexc[c >> 4]);
putchar(hexc[c & 0xf]);
}
static void text(MDB_val *v)
{
unsigned char *c, *end;
putchar(' ');
c = v->mv_data;
end = c + v->mv_size;
while (c < end) {
if (isprint(*c)) {
putchar(*c);
} else {
putchar('\\');
hex(*c);
}
c++;
}
putchar('\n');
}
static void byte(MDB_val *v)
{
unsigned char *c, *end;
putchar(' ');
c = v->mv_data;
end = c + v->mv_size;
while (c < end) {
hex(*c++);
}
putchar('\n');
}
/* Dump in BDB-compatible format */
static int dumpit(MDB_txn *txn, MDB_dbi dbi, char *name)
{
MDB_cursor *mc;
MDB_stat ms;
MDB_val key, data;
MDB_envinfo info;
unsigned int flags;
int rc, i;
rc = mdb_dbi_flags(txn, dbi, &flags);
if (rc) return rc;
rc = mdb_stat(txn, dbi, &ms);
if (rc) return rc;
rc = mdb_env_info(mdb_txn_env(txn), &info);
if (rc) return rc;
printf("VERSION=3\n");
printf("format=%s\n", mode & PRINT ? "print" : "bytevalue");
if (name)
printf("database=%s\n", name);
printf("type=btree\n");
printf("mapsize=%zu\n", info.me_mapsize);
if (info.me_mapaddr)
printf("mapaddr=%p\n", info.me_mapaddr);
printf("maxreaders=%u\n", info.me_maxreaders);
for (i=0; dbflags[i].bit; i++)
if (flags & dbflags[i].bit)
printf("%s=1\n", dbflags[i].name);
printf("db_pagesize=%d\n", ms.ms_psize);
printf("HEADER=END\n");
rc = mdb_cursor_open(txn, dbi, &mc);
if (rc) return rc;
while ((rc = mdb_cursor_get(mc, &key, &data, MDB_NEXT)) == MDB_SUCCESS) {
if (gotsig) {
rc = EINTR;
break;
}
if (mode & PRINT) {
text(&key);
text(&data);
} else {
byte(&key);
byte(&data);
}
}
printf("DATA=END\n");
if (rc == MDB_NOTFOUND)
rc = MDB_SUCCESS;
return rc;
}
static void usage(char *prog)
{
fprintf(stderr, "usage: %s [-V] [-f output] [-l] [-n] [-p] [-a|-s subdb] dbpath\n", prog);
exit(EXIT_FAILURE);
}
int main(int argc, char *argv[])
{
int i, rc;
MDB_env *env;
MDB_txn *txn;
MDB_dbi dbi;
char *prog = argv[0];
char *envname;
char *subname = NULL;
int alldbs = 0, envflags = 0, list = 0;
if (argc < 2) {
usage(prog);
}
/* -a: dump main DB and all subDBs
* -s: dump only the named subDB
* -n: use NOSUBDIR flag on env_open
* -p: use printable characters
* -f: write to file instead of stdout
* -V: print version and exit
* (default) dump only the main DB
*/
while ((i = getopt(argc, argv, "af:lnps:V")) != EOF) {
switch(i) {
case 'V':
printf("%s\n", MDB_VERSION_STRING);
exit(0);
break;
case 'l':
list = 1;
/*FALLTHROUGH*/;
case 'a':
if (subname)
usage(prog);
alldbs++;
break;
case 'f':
if (freopen(optarg, "w", stdout) == NULL) {
fprintf(stderr, "%s: %s: reopen: %s\n",
prog, optarg, strerror(errno));
exit(EXIT_FAILURE);
}
break;
case 'n':
envflags |= MDB_NOSUBDIR;
break;
case 'p':
mode |= PRINT;
break;
case 's':
if (alldbs)
usage(prog);
subname = optarg;
break;
default:
usage(prog);
}
}
if (optind != argc - 1)
usage(prog);
#ifdef SIGPIPE
signal(SIGPIPE, dumpsig);
#endif
#ifdef SIGHUP
signal(SIGHUP, dumpsig);
#endif
signal(SIGINT, dumpsig);
signal(SIGTERM, dumpsig);
envname = argv[optind];
rc = mdb_env_create(&env);
if (rc) {
fprintf(stderr, "mdb_env_create failed, error %d %s\n", rc, mdb_strerror(rc));
return EXIT_FAILURE;
}
if (alldbs || subname) {
mdb_env_set_maxdbs(env, 2);
}
rc = mdb_env_open(env, envname, envflags | MDB_RDONLY, 0664);
if (rc) {
fprintf(stderr, "mdb_env_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto env_close;
}
rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
if (rc) {
fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
goto env_close;
}
rc = mdb_open(txn, subname, 0, &dbi);
if (rc) {
fprintf(stderr, "mdb_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
if (alldbs) {
MDB_cursor *cursor;
MDB_val key;
int count = 0;
rc = mdb_cursor_open(txn, dbi, &cursor);
if (rc) {
fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP)) == 0) {
char *str;
MDB_dbi db2;
if (memchr(key.mv_data, '\0', key.mv_size))
continue;
count++;
str = malloc(key.mv_size+1);
memcpy(str, key.mv_data, key.mv_size);
str[key.mv_size] = '\0';
rc = mdb_open(txn, str, 0, &db2);
if (rc == MDB_SUCCESS) {
if (list) {
printf("%s\n", str);
list++;
} else {
rc = dumpit(txn, db2, str);
if (rc)
break;
}
mdb_close(env, db2);
}
free(str);
if (rc) continue;
}
mdb_cursor_close(cursor);
if (!count) {
fprintf(stderr, "%s: %s does not contain multiple databases\n", prog, envname);
rc = MDB_NOTFOUND;
} else if (rc == MDB_INCOMPATIBLE) {
/* LY: the record it not a named sub-db. */
rc = MDB_SUCCESS;
}
} else {
rc = dumpit(txn, dbi, subname);
}
if (rc && rc != MDB_NOTFOUND)
fprintf(stderr, "%s: %s: %s\n", prog, envname, mdb_strerror(rc));
mdb_close(env, dbi);
txn_abort:
mdb_txn_abort(txn);
env_close:
mdb_env_close(env);
return rc ? EXIT_FAILURE : EXIT_SUCCESS;
}

456
mdb_load.c
View File

@ -1,456 +0,0 @@
/* mdb_load.c - memory-mapped database load tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2011-2017 Howard Chu, Symas Corp.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <ctype.h>
#include <unistd.h>
#include "mdbx.h"
#define PRINT 1
#define NOHDR 2
static int mode;
static char *subname = NULL;
static size_t lineno;
static int version;
static int dbi_flags;
static char *prog;
static int Eof;
static MDB_envinfo info;
static MDB_val kbuf, dbuf;
#define STRLENOF(s) (sizeof(s)-1)
typedef struct flagbit {
int bit;
char *name;
int len;
} flagbit;
#define S(s) s, STRLENOF(s)
flagbit dbflags[] = {
{ MDB_REVERSEKEY, S("reversekey") },
{ MDB_DUPSORT, S("dupsort") },
{ MDB_INTEGERKEY, S("integerkey") },
{ MDB_DUPFIXED, S("dupfixed") },
{ MDB_INTEGERDUP, S("integerdup") },
{ MDB_REVERSEDUP, S("reversedup") },
{ 0, NULL, 0 }
};
static void readhdr(void)
{
char *ptr;
dbi_flags = 0;
while (fgets(dbuf.mv_data, dbuf.mv_size, stdin) != NULL) {
lineno++;
if (!strncmp(dbuf.mv_data, "db_pagesize=", STRLENOF("db_pagesize="))
|| !strncmp(dbuf.mv_data, "duplicates=", STRLENOF("duplicates="))) {
/* LY: silently ignore information fields. */
continue;
} else if (!strncmp(dbuf.mv_data, "VERSION=", STRLENOF("VERSION="))) {
version=atoi((char *)dbuf.mv_data+STRLENOF("VERSION="));
if (version > 3) {
fprintf(stderr, "%s: line %zd: unsupported VERSION %d\n",
prog, lineno, version);
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "HEADER=END", STRLENOF("HEADER=END"))) {
break;
} else if (!strncmp(dbuf.mv_data, "format=", STRLENOF("format="))) {
if (!strncmp((char *)dbuf.mv_data+STRLENOF("FORMAT="), "print", STRLENOF("print")))
mode |= PRINT;
else if (strncmp((char *)dbuf.mv_data+STRLENOF("FORMAT="), "bytevalue", STRLENOF("bytevalue"))) {
fprintf(stderr, "%s: line %zd: unsupported FORMAT %s\n",
prog, lineno, (char *)dbuf.mv_data+STRLENOF("FORMAT="));
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "database=", STRLENOF("database="))) {
ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
if (ptr) *ptr = '\0';
if (subname) free(subname);
subname = strdup((char *)dbuf.mv_data+STRLENOF("database="));
} else if (!strncmp(dbuf.mv_data, "type=", STRLENOF("type="))) {
if (strncmp((char *)dbuf.mv_data+STRLENOF("type="), "btree", STRLENOF("btree"))) {
fprintf(stderr, "%s: line %zd: unsupported type %s\n",
prog, lineno, (char *)dbuf.mv_data+STRLENOF("type="));
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "mapaddr=", STRLENOF("mapaddr="))) {
int i;
ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
if (ptr) *ptr = '\0';
i = sscanf((char *)dbuf.mv_data+STRLENOF("mapaddr="), "%p", &info.me_mapaddr);
if (i != 1) {
fprintf(stderr, "%s: line %zd: invalid mapaddr %s\n",
prog, lineno, (char *)dbuf.mv_data+STRLENOF("mapaddr="));
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "mapsize=", STRLENOF("mapsize="))) {
int i;
ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
if (ptr) *ptr = '\0';
i = sscanf((char *)dbuf.mv_data+STRLENOF("mapsize="), "%zu", &info.me_mapsize);
if (i != 1) {
fprintf(stderr, "%s: line %zd: invalid mapsize %s\n",
prog, lineno, (char *)dbuf.mv_data+STRLENOF("mapsize="));
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "maxreaders=", STRLENOF("maxreaders="))) {
int i;
ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
if (ptr) *ptr = '\0';
i = sscanf((char *)dbuf.mv_data+STRLENOF("maxreaders="), "%u", &info.me_maxreaders);
if (i != 1) {
fprintf(stderr, "%s: line %zd: invalid maxreaders %s\n",
prog, lineno, (char *)dbuf.mv_data+STRLENOF("maxreaders="));
exit(EXIT_FAILURE);
}
} else {
int i;
for (i=0; dbflags[i].bit; i++) {
if (!strncmp(dbuf.mv_data, dbflags[i].name, dbflags[i].len) &&
((char *)dbuf.mv_data)[dbflags[i].len] == '=') {
if (((char *)dbuf.mv_data)[dbflags[i].len+1] == '1')
dbi_flags |= dbflags[i].bit;
break;
}
}
if (!dbflags[i].bit) {
ptr = memchr(dbuf.mv_data, '=', dbuf.mv_size);
if (!ptr) {
fprintf(stderr, "%s: line %zd: unexpected format\n",
prog, lineno);
exit(EXIT_FAILURE);
} else {
*ptr = '\0';
fprintf(stderr, "%s: line %zd: unrecognized keyword ignored: %s\n",
prog, lineno, (char *)dbuf.mv_data);
}
}
}
}
}
static void badend(void)
{
fprintf(stderr, "%s: line %zd: unexpected end of input\n",
prog, lineno);
}
static int unhex(unsigned char *c2)
{
int x, c;
x = *c2++ & 0x4f;
if (x & 0x40)
x -= 55;
c = x << 4;
x = *c2 & 0x4f;
if (x & 0x40)
x -= 55;
c |= x;
return c;
}
static int readline(MDB_val *out, MDB_val *buf)
{
unsigned char *c1, *c2, *end;
size_t len, l2;
int c;
if (!(mode & NOHDR)) {
c = fgetc(stdin);
if (c == EOF) {
Eof = 1;
return EOF;
}
if (c != ' ') {
lineno++;
if (fgets(buf->mv_data, buf->mv_size, stdin) == NULL) {
badend:
Eof = 1;
badend();
return EOF;
}
if (c == 'D' && !strncmp(buf->mv_data, "ATA=END", STRLENOF("ATA=END")))
return EOF;
goto badend;
}
}
if (fgets(buf->mv_data, buf->mv_size, stdin) == NULL) {
Eof = 1;
return EOF;
}
lineno++;
c1 = buf->mv_data;
len = strlen((char *)c1);
l2 = len;
/* Is buffer too short? */
while (c1[len-1] != '\n') {
buf->mv_data = realloc(buf->mv_data, buf->mv_size*2);
if (!buf->mv_data) {
Eof = 1;
fprintf(stderr, "%s: line %zd: out of memory, line too long\n",
prog, lineno);
return EOF;
}
c1 = buf->mv_data;
c1 += l2;
if (fgets((char *)c1, buf->mv_size+1, stdin) == NULL) {
Eof = 1;
badend();
return EOF;
}
buf->mv_size *= 2;
len = strlen((char *)c1);
l2 += len;
}
c1 = c2 = buf->mv_data;
len = l2;
c1[--len] = '\0';
end = c1 + len;
if (mode & PRINT) {
while (c2 < end) {
if (*c2 == '\\') {
if (c2[1] == '\\') {
c1++; c2 += 2;
} else {
if (c2+3 > end || !isxdigit(c2[1]) || !isxdigit(c2[2])) {
Eof = 1;
badend();
return EOF;
}
*c1++ = unhex(++c2);
c2 += 2;
}
} else {
/* copies are redundant when no escapes were used */
*c1++ = *c2++;
}
}
} else {
/* odd length not allowed */
if (len & 1) {
Eof = 1;
badend();
return EOF;
}
while (c2 < end) {
if (!isxdigit(*c2) || !isxdigit(c2[1])) {
Eof = 1;
badend();
return EOF;
}
*c1++ = unhex(c2);
c2 += 2;
}
}
c2 = out->mv_data = buf->mv_data;
out->mv_size = c1 - c2;
return 0;
}
static void usage(void)
{
fprintf(stderr, "usage: %s [-V] [-f input] [-n] [-s name] [-N] [-T] dbpath\n", prog);
exit(EXIT_FAILURE);
}
int main(int argc, char *argv[])
{
int i, rc;
MDB_env *env;
MDB_txn *txn;
MDB_cursor *mc;
MDB_dbi dbi;
char *envname;
int envflags = 0, putflags = 0;
prog = argv[0];
if (argc < 2) {
usage();
}
/* -f: load file instead of stdin
* -n: use NOSUBDIR flag on env_open
* -s: load into named subDB
* -N: use NOOVERWRITE on puts
* -T: read plaintext
* -V: print version and exit
*/
while ((i = getopt(argc, argv, "f:ns:NTV")) != EOF) {
switch(i) {
case 'V':
printf("%s\n", MDB_VERSION_STRING);
exit(0);
break;
case 'f':
if (freopen(optarg, "r", stdin) == NULL) {
fprintf(stderr, "%s: %s: reopen: %s\n",
prog, optarg, strerror(errno));
exit(EXIT_FAILURE);
}
break;
case 'n':
envflags |= MDB_NOSUBDIR;
break;
case 's':
subname = strdup(optarg);
break;
case 'N':
putflags = MDB_NOOVERWRITE|MDB_NODUPDATA;
break;
case 'T':
mode |= NOHDR | PRINT;
break;
default:
usage();
}
}
if (optind != argc - 1)
usage();
dbuf.mv_size = 4096;
dbuf.mv_data = malloc(dbuf.mv_size);
if (!(mode & NOHDR))
readhdr();
envname = argv[optind];
rc = mdb_env_create(&env);
if (rc) {
fprintf(stderr, "mdb_env_create failed, error %d %s\n", rc, mdb_strerror(rc));
return EXIT_FAILURE;
}
mdb_env_set_maxdbs(env, 2);
if (info.me_maxreaders)
mdb_env_set_maxreaders(env, info.me_maxreaders);
if (info.me_mapsize)
mdb_env_set_mapsize(env, info.me_mapsize);
if (info.me_mapaddr)
envflags |= MDB_FIXEDMAP;
rc = mdb_env_open(env, envname, envflags, 0664);
if (rc) {
fprintf(stderr, "mdb_env_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto env_close;
}
kbuf.mv_size = mdb_env_get_maxkeysize(env) * 2 + 2;
kbuf.mv_data = malloc(kbuf.mv_size);
while(!Eof) {
MDB_val key, data;
int batch = 0;
rc = mdb_txn_begin(env, NULL, 0, &txn);
if (rc) {
fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
goto env_close;
}
rc = mdb_open(txn, subname, dbi_flags|MDB_CREATE, &dbi);
if (rc) {
fprintf(stderr, "mdb_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
rc = mdb_cursor_open(txn, dbi, &mc);
if (rc) {
fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
while(1) {
rc = readline(&key, &kbuf);
if (rc) /* rc == EOF */
break;
rc = readline(&data, &dbuf);
if (rc) {
fprintf(stderr, "%s: line %zd: failed to read key value\n", prog, lineno);
goto txn_abort;
}
rc = mdb_cursor_put(mc, &key, &data, putflags);
if (rc == MDB_KEYEXIST && putflags)
continue;
if (rc) {
fprintf(stderr, "mdb_cursor_put failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
batch++;
if (batch == 100) {
rc = mdb_txn_commit(txn);
if (rc) {
fprintf(stderr, "%s: line %zd: txn_commit: %s\n",
prog, lineno, mdb_strerror(rc));
goto env_close;
}
rc = mdb_txn_begin(env, NULL, 0, &txn);
if (rc) {
fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
goto env_close;
}
rc = mdb_cursor_open(txn, dbi, &mc);
if (rc) {
fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
batch = 0;
}
}
rc = mdb_txn_commit(txn);
txn = NULL;
if (rc) {
fprintf(stderr, "%s: line %zd: txn_commit: %s\n",
prog, lineno, mdb_strerror(rc));
goto env_close;
}
mdb_dbi_close(env, dbi);
if(!(mode & NOHDR))
readhdr();
}
txn_abort:
mdb_txn_abort(txn);
env_close:
mdb_env_close(env);
return rc ? EXIT_FAILURE : EXIT_SUCCESS;
}

299
mdb_stat.c
View File

@ -1,299 +0,0 @@
/* mdb_stat.c - memory-mapped database status tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2011-2017 Howard Chu, Symas Corp.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include "mdbx.h"
static void prstat(MDBX_stat *ms)
{
#if 0
printf(" Page size: %u\n", ms->base.ms_psize);
#endif
printf(" Tree depth: %u\n", ms->base.ms_depth);
printf(" Branch pages: %zu\n", ms->base.ms_branch_pages);
printf(" Leaf pages: %zu\n", ms->base.ms_leaf_pages);
printf(" Overflow pages: %zu\n", ms->base.ms_overflow_pages);
printf(" Entries: %zu\n", ms->base.ms_entries);
}
static void usage(char *prog)
{
fprintf(stderr, "usage: %s [-V] [-n] [-e] [-r[r]] [-f[f[f]]] [-a|-s subdb] dbpath\n", prog);
exit(EXIT_FAILURE);
}
int main(int argc, char *argv[])
{
int i, rc;
MDB_env *env;
MDB_txn *txn;
MDB_dbi dbi;
MDBX_stat mst;
MDBX_envinfo mei;
char *prog = argv[0];
char *envname;
char *subname = NULL;
int alldbs = 0, envinfo = 0, envflags = 0, freinfo = 0, rdrinfo = 0;
if (argc < 2) {
usage(prog);
}
/* -a: print stat of main DB and all subDBs
* -s: print stat of only the named subDB
* -e: print env info
* -f: print freelist info
* -r: print reader info
* -n: use NOSUBDIR flag on env_open
* -V: print version and exit
* (default) print stat of only the main DB
*/
while ((i = getopt(argc, argv, "Vaefnrs:")) != EOF) {
switch(i) {
case 'V':
printf("%s\n", MDB_VERSION_STRING);
exit(0);
break;
case 'a':
if (subname)
usage(prog);
alldbs++;
break;
case 'e':
envinfo++;
break;
case 'f':
freinfo++;
break;
case 'n':
envflags |= MDB_NOSUBDIR;
break;
case 'r':
rdrinfo++;
break;
case 's':
if (alldbs)
usage(prog);
subname = optarg;
break;
default:
usage(prog);
}
}
if (optind != argc - 1)
usage(prog);
envname = argv[optind];
rc = mdb_env_create(&env);
if (rc) {
fprintf(stderr, "mdb_env_create failed, error %d %s\n", rc, mdb_strerror(rc));
return EXIT_FAILURE;
}
if (alldbs || subname) {
mdb_env_set_maxdbs(env, 4);
}
rc = mdb_env_open(env, envname, envflags | MDB_RDONLY, 0664);
if (rc) {
fprintf(stderr, "mdb_env_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto env_close;
}
if (envinfo) {
(void)mdbx_env_stat(env, &mst, sizeof(mst));
(void)mdbx_env_info(env, &mei, sizeof(mei));
printf("Environment Info\n");
printf(" Map address: %p\n", mei.base.me_mapaddr);
printf(" Map size: %zu\n", mei.base.me_mapsize);
printf(" Page size: %u\n", mst.base.ms_psize);
printf(" Max pages: %zu\n", mei.base.me_mapsize / mst.base.ms_psize);
printf(" Number of pages used: %zu\n", mei.base.me_last_pgno+1);
printf(" Last transaction ID: %zu\n", mei.base.me_last_txnid);
printf(" Tail transaction ID: %zu (%zi)\n",
mei.me_tail_txnid, mei.me_tail_txnid - mei.base.me_last_txnid);
printf(" Max readers: %u\n", mei.base.me_maxreaders);
printf(" Number of readers used: %u\n", mei.base.me_numreaders);
} else {
/* LY: zap warnings from gcc */
memset(&mst, 0, sizeof(mst));
memset(&mei, 0, sizeof(mei));
}
if (rdrinfo) {
printf("Reader Table Status\n");
rc = mdb_reader_list(env, (MDB_msg_func *)fputs, stdout);
if (rdrinfo > 1) {
int dead;
mdb_reader_check(env, &dead);
printf(" %d stale readers cleared.\n", dead);
rc = mdb_reader_list(env, (MDB_msg_func *)fputs, stdout);
}
if (!(subname || alldbs || freinfo))
goto env_close;
}
rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
if (rc) {
fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
goto env_close;
}
if (freinfo) {
MDB_cursor *cursor;
MDB_val key, data;
size_t pages = 0, *iptr;
size_t reclaimable = 0;
printf("Freelist Status\n");
dbi = 0;
rc = mdb_cursor_open(txn, dbi, &cursor);
if (rc) {
fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
rc = mdbx_stat(txn, dbi, &mst, sizeof(mst));
if (rc) {
fprintf(stderr, "mdb_stat failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
prstat(&mst);
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
iptr = data.mv_data;
pages += *iptr;
if (envinfo && mei.me_tail_txnid > *(size_t *)key.mv_data)
reclaimable += *iptr;
if (freinfo > 1) {
char *bad = "";
size_t pg, prev;
ssize_t i, j, span = 0;
j = *iptr++;
for (i = j, prev = 1; --i >= 0; ) {
pg = iptr[i];
if (pg <= prev)
bad = " [bad sequence]";
prev = pg;
pg += span;
for (; i >= span && iptr[i-span] == pg; span++, pg++) ;
}
printf(" Transaction %zu, %zd pages, maxspan %zd%s\n",
*(size_t *)key.mv_data, j, span, bad);
if (freinfo > 2) {
for (--j; j >= 0; ) {
pg = iptr[j];
for (span=1; --j >= 0 && iptr[j] == pg+span; span++) ;
if (span>1)
printf(" %9zu[%zd]\n", pg, span);
else
printf(" %9zu\n", pg);
}
}
}
}
mdb_cursor_close(cursor);
if (envinfo) {
size_t value = mei.base.me_mapsize / mst.base.ms_psize;
double percent = value / 100.0;
printf("Page Allocation Info\n");
printf(" Max pages: %9zu 100%%\n", value);
value = mei.base.me_last_pgno+1;
printf(" Number of pages used: %zu %.1f%%\n", value, value / percent);
value = mei.base.me_mapsize / mst.base.ms_psize - (mei.base.me_last_pgno+1);
printf(" Remained: %zu %.1f%%\n", value, value / percent);
value = mei.base.me_last_pgno+1 - pages;
printf(" Used now: %zu %.1f%%\n", value, value / percent);
value = pages;
printf(" Unallocated: %zu %.1f%%\n", value, value / percent);
value = pages - reclaimable;
printf(" Detained: %zu %.1f%%\n", value, value / percent);
value = reclaimable;
printf(" Reclaimable: %zu %.1f%%\n", value, value / percent);
value = mei.base.me_mapsize / mst.base.ms_psize - (mei.base.me_last_pgno+1) + reclaimable;
printf(" Available: %zu %.1f%%\n", value, value / percent);
} else
printf(" Free pages: %zu\n", pages);
}
rc = mdb_open(txn, subname, 0, &dbi);
if (rc) {
fprintf(stderr, "mdb_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
rc = mdbx_stat(txn, dbi, &mst, sizeof(mst));
if (rc) {
fprintf(stderr, "mdb_stat failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
printf("Status of %s\n", subname ? subname : "Main DB");
prstat(&mst);
if (alldbs) {
MDB_cursor *cursor;
MDB_val key;
rc = mdb_cursor_open(txn, dbi, &cursor);
if (rc) {
fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP)) == 0) {
char *str;
MDB_dbi db2;
if (memchr(key.mv_data, '\0', key.mv_size))
continue;
str = malloc(key.mv_size+1);
memcpy(str, key.mv_data, key.mv_size);
str[key.mv_size] = '\0';
rc = mdb_open(txn, str, 0, &db2);
if (rc == MDB_SUCCESS)
printf("Status of %s\n", str);
free(str);
if (rc) continue;
rc = mdbx_stat(txn, db2, &mst, sizeof(mst));
if (rc) {
fprintf(stderr, "mdb_stat failed, error %d %s\n", rc, mdb_strerror(rc));
goto txn_abort;
}
prstat(&mst);
mdb_close(env, db2);
}
mdb_cursor_close(cursor);
}
if (rc == MDB_NOTFOUND)
rc = MDB_SUCCESS;
mdb_close(env, dbi);
txn_abort:
mdb_txn_abort(txn);
env_close:
mdb_env_close(env);
return rc ? EXIT_FAILURE : EXIT_SUCCESS;
}

11047
mdbx.c
View File

File diff suppressed because it is too large Load Diff

1773
mdbx.h
View File

File diff suppressed because it is too large Load Diff

979
mdbx_chk.c Normal file
View File

@ -0,0 +1,979 @@
/* mdbx_chk.c - memory-mapped database check tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2015,2016 Peter-Service R&D LLC.
*
* This file is part of libmdbx.
*
* libmdbx is free software; you can redistribute it and/or modify it under
* the terms of the GNU Affero General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* (at your option) any later version.
*
* libmdbx is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
*/
#include <ctype.h>
#include <errno.h>
#include <malloc.h>
#include <signal.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include "mdbx.h"
#include "midl.h"
typedef struct flagbit {
int bit;
char *name;
} flagbit;
flagbit dbflags[] = {{MDB_DUPSORT, "dupsort"},
{MDB_INTEGERKEY, "integerkey"},
{MDB_REVERSEKEY, "reversekey"},
{MDB_DUPFIXED, "dupfixed"},
{MDB_REVERSEDUP, "reversedup"},
{MDB_INTEGERDUP, "integerdup"},
{0, NULL}};
static volatile sig_atomic_t gotsignal;
static void signal_handler(int sig) {
(void)sig;
gotsignal = 1;
}
#define MAX_DBI 32768
#define EXIT_INTERRUPTED (EXIT_FAILURE + 4)
#define EXIT_FAILURE_SYS (EXIT_FAILURE + 3)
#define EXIT_FAILURE_MDB (EXIT_FAILURE + 2)
#define EXIT_FAILURE_CHECK_MAJOR (EXIT_FAILURE + 1)
#define EXIT_FAILURE_CHECK_MINOR EXIT_FAILURE
struct {
const char *dbi_names[MAX_DBI];
size_t dbi_pages[MAX_DBI];
size_t dbi_empty_pages[MAX_DBI];
size_t dbi_payload_bytes[MAX_DBI];
size_t dbi_lost_bytes[MAX_DBI];
short *pagemap;
size_t total_payload_bytes;
size_t pgcount;
} walk;
static __attribute__((constructor)) void init_walk(void) {
walk.dbi_names[0] = "@gc";
}
size_t total_unused_bytes;
int exclusive = 2;
MDB_env *env;
MDB_txn *txn, *locktxn;
MDBX_envinfo info;
MDBX_stat stat;
size_t maxkeysize, reclaimable_pages, freedb_pages, lastpgno;
size_t userdb_count, skipped_subdb;
unsigned verbose, quiet;
const char *only_subdb;
struct problem {
struct problem *pr_next;
size_t count;
const char *caption;
};
struct problem *problems_list;
size_t total_problems;
static void __attribute__((format(printf, 1, 2))) print(const char *msg, ...) {
if (!quiet) {
va_list args;
fflush(stderr);
va_start(args, msg);
vfprintf(stdout, msg, args);
va_end(args);
}
}
static void __attribute__((format(printf, 1, 2))) error(const char *msg, ...) {
total_problems++;
if (!quiet) {
va_list args;
fflush(stdout);
va_start(args, msg);
vfprintf(stderr, msg, args);
va_end(args);
fflush(NULL);
}
}
static void pagemap_cleanup(void) {
int i;
for (i = 1; i < MAX_DBI; ++i) {
if (walk.dbi_names[i]) {
free((void *)walk.dbi_names[i]);
walk.dbi_names[i] = NULL;
}
}
free(walk.pagemap);
walk.pagemap = NULL;
}
static int pagemap_lookup_dbi(const char *dbi) {
static int last;
int i;
if (last > 0 && strcmp(walk.dbi_names[last], dbi) == 0)
return last;
for (i = 1; walk.dbi_names[i] && last < MAX_DBI; ++i)
if (strcmp(walk.dbi_names[i], dbi) == 0)
return last = i;
if (i == MAX_DBI)
return -1;
walk.dbi_names[i] = strdup(dbi);
if (verbose > 1) {
print(" - found '%s' area\n", dbi);
fflush(NULL);
}
return last = i;
}
static void problem_add(const char *object, size_t entry_number,
const char *msg, const char *extra, ...) {
total_problems++;
if (!quiet) {
int need_fflush = 0;
struct problem *p;
for (p = problems_list; p; p = p->pr_next)
if (p->caption == msg)
break;
if (!p) {
p = calloc(1, sizeof(*p));
p->caption = msg;
p->pr_next = problems_list;
problems_list = p;
need_fflush = 1;
}
p->count++;
if (verbose > 1) {
print(" %s #%zu: %s", object, entry_number, msg);
if (extra) {
va_list args;
printf(" (");
va_start(args, extra);
vfprintf(stdout, extra, args);
va_end(args);
printf(")");
}
printf("\n");
if (need_fflush)
fflush(NULL);
}
}
}
static struct problem *problems_push() {
struct problem *p = problems_list;
problems_list = NULL;
return p;
}
static size_t problems_pop(struct problem *list) {
size_t count = 0;
if (problems_list) {
int i;
print(" - problems: ");
for (i = 0; problems_list; ++i) {
struct problem *p = problems_list->pr_next;
count += problems_list->count;
print("%s%s (%zu)", i ? ", " : "", problems_list->caption,
problems_list->count);
free(problems_list);
problems_list = p;
}
print("\n");
fflush(NULL);
}
problems_list = list;
return count;
}
static int pgvisitor(size_t pgno, unsigned pgnumber, void *ctx, const char *dbi,
const char *type, int nentries, int payload_bytes,
int header_bytes, int unused_bytes) {
(void)ctx;
if (type) {
size_t page_bytes = payload_bytes + header_bytes + unused_bytes;
size_t page_size = pgnumber * stat.ms_psize;
int index = pagemap_lookup_dbi(dbi);
if (index < 0)
return ENOMEM;
if (verbose > 2 && (!only_subdb || strcmp(only_subdb, dbi) == 0)) {
if (pgnumber == 1)
print(" %s-page %zu", type, pgno);
else
print(" %s-span %zu[%u]", type, pgno, pgnumber);
print(" of %s: header %i, payload %i, unused %i\n", dbi, header_bytes,
payload_bytes, unused_bytes);
}
walk.pgcount += pgnumber;
if (unused_bytes < 0 || (size_t)unused_bytes > page_size)
problem_add("page", pgno, "illegal unused-bytes", "%zu < %i < %zu", 0,
unused_bytes, stat.ms_psize);
if (header_bytes < (int)sizeof(long) ||
(size_t)header_bytes >= stat.ms_psize - sizeof(long))
problem_add("page", pgno, "illegal header-length", "%zu < %i < %zu",
sizeof(long), header_bytes, stat.ms_psize - sizeof(long));
if (payload_bytes < 1) {
if (nentries > 1) {
problem_add("page", pgno, "zero size-of-entry",
"payload %i bytes, %i entries", payload_bytes, nentries);
if ((size_t)header_bytes + unused_bytes < page_size) {
/* LY: hush a misuse error */
page_bytes = page_size;
}
} else {
problem_add("page", pgno, "empty", "payload %i bytes, %i entries",
payload_bytes, nentries);
walk.dbi_empty_pages[index] += 1;
}
}
if (page_bytes != page_size) {
problem_add("page", pgno, "misused", "%zu != %zu (%ih + %ip + %iu)",
page_size, page_bytes, header_bytes, payload_bytes,
unused_bytes);
if (page_size > page_bytes)
walk.dbi_lost_bytes[index] += page_size - page_bytes;
} else {
walk.dbi_payload_bytes[index] += payload_bytes + header_bytes;
walk.total_payload_bytes += payload_bytes + header_bytes;
}
if (pgnumber) {
do {
if (pgno >= lastpgno)
problem_add("page", pgno, "wrong page-no", "%zu > %zi", pgno,
lastpgno);
else if (walk.pagemap[pgno])
problem_add("page", pgno, "already used", "in %s",
walk.dbi_names[walk.pagemap[pgno]]);
else {
walk.pagemap[pgno] = index;
walk.dbi_pages[index] += 1;
}
++pgno;
} while (--pgnumber);
}
}
return gotsignal ? EINTR : MDB_SUCCESS;
}
typedef int(visitor)(size_t record_number, MDB_val *key, MDB_val *data);
static int process_db(MDB_dbi dbi, char *name, visitor *handler, int silent);
static int handle_userdb(size_t record_number, MDB_val *key, MDB_val *data) {
(void)record_number;
(void)key;
(void)data;
return MDB_SUCCESS;
}
static int handle_freedb(size_t record_number, MDB_val *key, MDB_val *data) {
char *bad = "";
size_t pg, prev;
ssize_t i, number, span = 0;
size_t *iptr = data->mv_data, txnid = *(size_t *)key->mv_data;
if (key->mv_size != sizeof(txnid))
problem_add("entry", record_number, "wrong txn-id size", "key-size %zi",
key->mv_size);
else if (txnid < 1 || txnid > info.me_last_txnid)
problem_add("entry", record_number, "wrong txn-id", "%zu", txnid);
if (data->mv_size < sizeof(size_t) || data->mv_size % sizeof(size_t))
problem_add("entry", record_number, "wrong idl size", "%zu", data->mv_size);
else {
number = *iptr++;
if (number >= MDB_IDL_UM_MAX)
problem_add("entry", record_number, "wrong idl length", "%zi", number);
else if ((number + 1) * sizeof(size_t) != data->mv_size)
problem_add("entry", record_number, "mismatch idl length", "%zi != %zu",
number * sizeof(size_t), data->mv_size);
else {
freedb_pages += number;
if (info.me_tail_txnid > txnid)
reclaimable_pages += number;
for (i = number, prev = 1; --i >= 0;) {
pg = iptr[i];
if (pg < 2 /* META_PAGE */ || pg > info.me_last_pgno)
problem_add("entry", record_number, "wrong idl entry",
"2 < %zi < %zi", pg, info.me_last_pgno);
else if (pg <= prev) {
bad = " [bad sequence]";
problem_add("entry", record_number, "bad sequence", "%zi <= %zi", pg,
prev);
}
prev = pg;
pg += span;
for (; i >= span && iptr[i - span] == pg; span++, pg++)
;
}
if (verbose > 2 && !only_subdb) {
print(" transaction %zu, %zd pages, maxspan %zd%s\n",
*(size_t *)key->mv_data, number, span, bad);
if (verbose > 3) {
int j = number - 1;
while (j >= 0) {
pg = iptr[j];
for (span = 1; --j >= 0 && iptr[j] == pg + span; span++)
;
if (span > 1)
print(" %9zu[%zd]\n", pg, span);
else
print(" %9zu\n", pg);
}
}
}
}
}
return MDB_SUCCESS;
}
static int handle_maindb(size_t record_number, MDB_val *key, MDB_val *data) {
char *name;
int rc;
size_t i;
name = key->mv_data;
for (i = 0; i < key->mv_size; ++i) {
if (name[i] < ' ')
return handle_userdb(record_number, key, data);
}
name = malloc(key->mv_size + 1);
memcpy(name, key->mv_data, key->mv_size);
name[key->mv_size] = '\0';
userdb_count++;
rc = process_db(-1, name, handle_userdb, 0);
free(name);
if (rc != MDB_INCOMPATIBLE)
return rc;
return handle_userdb(record_number, key, data);
}
static int process_db(MDB_dbi dbi, char *name, visitor *handler, int silent) {
MDB_cursor *mc;
MDBX_stat ms;
MDB_val key, data;
MDB_val prev_key, prev_data;
unsigned flags;
int rc, i;
struct problem *saved_list;
size_t problems_count;
unsigned record_count = 0, dups = 0;
size_t key_bytes = 0, data_bytes = 0;
if (0 > (int)dbi) {
rc = mdbx_dbi_open(txn, name, 0, &dbi);
if (rc) {
if (!name ||
rc !=
MDB_INCOMPATIBLE) /* LY: mainDB's record is not a user's DB. */ {
error(" - mdbx_open '%s' failed, error %d %s\n", name ? name : "main",
rc, mdbx_strerror(rc));
}
return rc;
}
}
if (dbi >= 2 /* CORE_DBS */ && name && only_subdb &&
strcmp(only_subdb, name)) {
if (verbose) {
print("Skip processing '%s'...\n", name);
fflush(NULL);
}
skipped_subdb++;
return MDB_SUCCESS;
}
if (!silent && verbose) {
print("Processing '%s'...\n", name ? name : "main");
fflush(NULL);
}
rc = mdbx_dbi_flags(txn, dbi, &flags);
if (rc) {
error(" - mdbx_dbi_flags failed, error %d %s\n", rc, mdbx_strerror(rc));
return rc;
}
rc = mdbx_stat(txn, dbi, &ms, sizeof(ms));
if (rc) {
error(" - mdbx_stat failed, error %d %s\n", rc, mdbx_strerror(rc));
return rc;
}
if (!silent && verbose) {
print(" - dbi-id %d, flags:", dbi);
if (!flags)
print(" none");
else {
for (i = 0; dbflags[i].bit; i++)
if (flags & dbflags[i].bit)
print(" %s", dbflags[i].name);
}
print(" (0x%02X)\n", flags);
if (verbose > 1) {
print(" - page size %u, entries %zu\n", ms.ms_psize, ms.ms_entries);
print(" - b-tree depth %u, pages: branch %zu, leaf %zu, overflow %zu\n",
ms.ms_depth, ms.ms_branch_pages, ms.ms_leaf_pages,
ms.ms_overflow_pages);
}
}
rc = mdbx_cursor_open(txn, dbi, &mc);
if (rc) {
error(" - mdbx_cursor_open failed, error %d %s\n", rc, mdbx_strerror(rc));
return rc;
}
saved_list = problems_push();
prev_key.mv_data = NULL;
prev_data.mv_size = 0;
rc = mdbx_cursor_get(mc, &key, &data, MDB_FIRST);
while (rc == MDB_SUCCESS) {
if (gotsignal) {
print(" - interrupted by signal\n");
fflush(NULL);
rc = EINTR;
goto bailout;
}
if (key.mv_size > maxkeysize) {
problem_add("entry", record_count, "key length exceeds max-key-size",
"%zu > %zu", key.mv_size, maxkeysize);
} else if ((flags & MDB_INTEGERKEY) && key.mv_size != sizeof(size_t) &&
key.mv_size != sizeof(int)) {
problem_add("entry", record_count, "wrong key length", "%zu != %zu",
key.mv_size, sizeof(size_t));
}
if ((flags & MDB_INTEGERDUP) && data.mv_size != sizeof(size_t) &&
data.mv_size != sizeof(int)) {
problem_add("entry", record_count, "wrong data length", "%zu != %zu",
data.mv_size, sizeof(size_t));
}
if (prev_key.mv_data) {
if ((flags & MDB_DUPFIXED) && prev_data.mv_size != data.mv_size) {
problem_add("entry", record_count, "different data length",
"%zu != %zu", prev_data.mv_size, data.mv_size);
}
int cmp = mdbx_cmp(txn, dbi, &prev_key, &key);
if (cmp > 0) {
problem_add("entry", record_count, "broken ordering of entries", NULL);
} else if (cmp == 0) {
++dups;
if (!(flags & MDB_DUPSORT))
problem_add("entry", record_count, "duplicated entries", NULL);
else if (flags & MDB_INTEGERDUP) {
cmp = mdbx_dcmp(txn, dbi, &prev_data, &data);
if (cmp > 0)
problem_add("entry", record_count,
"broken ordering of multi-values", NULL);
}
}
} else if (verbose) {
if (flags & MDB_INTEGERKEY)
print(" - fixed key-size %zu\n", key.mv_size);
if (flags & (MDB_INTEGERDUP | MDB_DUPFIXED))
print(" - fixed data-size %zu\n", data.mv_size);
}
if (handler) {
rc = handler(record_count, &key, &data);
if (rc)
goto bailout;
}
record_count++;
key_bytes += key.mv_size;
data_bytes += data.mv_size;
prev_key = key;
prev_data = data;
rc = mdbx_cursor_get(mc, &key, &data, MDB_NEXT);
}
if (rc != MDB_NOTFOUND)
error(" - mdbx_cursor_get failed, error %d %s\n", rc, mdbx_strerror(rc));
else
rc = 0;
if (record_count != ms.ms_entries)
problem_add("entry", record_count, "differentent number of entries",
"%zu != %zu", record_count, ms.ms_entries);
bailout:
problems_count = problems_pop(saved_list);
if (!silent && verbose) {
print(" - summary: %u records, %u dups, %zu key's bytes, %zu data's "
"bytes, %zu problems\n",
record_count, dups, key_bytes, data_bytes, problems_count);
fflush(NULL);
}
mdbx_cursor_close(mc);
return rc || problems_count;
}
static void usage(char *prog) {
fprintf(stderr,
"usage: %s dbpath [-V] [-v] [-n] [-q] [-w] [-c] [-d] [-s subdb]\n"
" -V\t\tshow version\n"
" -v\t\tmore verbose, could be used multiple times\n"
" -n\t\tNOSUBDIR mode for open\n"
" -q\t\tbe quiet\n"
" -w\t\tlock DB for writing while checking\n"
" -d\t\tdisable page-by-page traversal of b-tree\n"
" -s subdb\tprocess a specific subdatabase only\n"
" -c\t\tforce cooperative mode (don't try exclusive)\n",
prog);
exit(EXIT_INTERRUPTED);
}
const char *meta_synctype(size_t sign) {
switch (sign) {
case 0:
return "no-sync/legacy";
case 1:
return "weak";
default:
return "steady";
}
}
int meta_lt(size_t txn1, size_t sign1, size_t txn2, size_t sign2) {
return ((sign1 > 1) == (sign2 > 1)) ? txn1 < txn2 : txn2 && sign2 > 1;
}
int main(int argc, char *argv[]) {
int i, rc;
char *prog = argv[0];
char *envname;
int envflags = MDB_RDONLY;
int problems_maindb = 0, problems_freedb = 0, problems_meta = 0;
int dont_traversal = 0;
size_t n;
struct timespec timestamp_start, timestamp_finish;
double elapsed;
atexit(pagemap_cleanup);
if (clock_gettime(CLOCK_MONOTONIC, &timestamp_start)) {
rc = errno;
error("clock_gettime failed, error %d %s\n", rc, mdbx_strerror(rc));
return EXIT_FAILURE_SYS;
}
if (argc < 2) {
usage(prog);
}
while ((i = getopt(argc, argv, "Vvqnwcds:")) != EOF) {
switch (i) {
case 'V':
printf("%s\n", MDB_VERSION_STRING);
exit(EXIT_SUCCESS);
break;
case 'v':
verbose++;
break;
case 'q':
quiet = 1;
break;
case 'n':
envflags |= MDB_NOSUBDIR;
break;
case 'w':
envflags &= ~MDB_RDONLY;
break;
case 'c':
exclusive = 0;
break;
case 'd':
dont_traversal = 1;
break;
case 's':
if (only_subdb && strcmp(only_subdb, optarg))
usage(prog);
only_subdb = optarg;
break;
default:
usage(prog);
}
}
if (optind != argc - 1)
usage(prog);
#ifdef SIGPIPE
signal(SIGPIPE, signal_handler);
#endif
#ifdef SIGHUP
signal(SIGHUP, signal_handler);
#endif
signal(SIGINT, signal_handler);
signal(SIGTERM, signal_handler);
envname = argv[optind];
print("Running mdbx_chk for '%s' in %s mode...\n", envname,
(envflags & MDB_RDONLY) ? "read-only" : "write-lock");
fflush(NULL);
rc = mdbx_env_create(&env);
if (rc) {
error("mdbx_env_create failed, error %d %s\n", rc, mdbx_strerror(rc));
return rc < 0 ? EXIT_FAILURE_MDB : EXIT_FAILURE_SYS;
}
rc = mdbx_env_get_maxkeysize(env);
if (rc < 0) {
error("mdbx_env_get_maxkeysize failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto bailout;
}
maxkeysize = rc;
rc = mdbx_env_set_maxdbs(env, MAX_DBI);
if (rc < 0) {
error("mdbx_env_set_maxdbs failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
rc = mdbx_env_open_ex(env, envname, envflags, 0664, &exclusive);
if (rc) {
error("mdbx_env_open failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
if (verbose)
print(" - %s mode\n", exclusive ? "monopolistic" : "cooperative");
if (!(envflags & MDB_RDONLY)) {
rc = mdbx_txn_begin(env, NULL, 0, &locktxn);
if (rc) {
error("mdbx_txn_begin(lock-write) failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto bailout;
}
}
rc = mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn);
if (rc) {
error("mdbx_txn_begin(read-only) failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto bailout;
}
rc = mdbx_env_info(env, &info, sizeof(info));
if (rc) {
error("mdbx_env_info failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
rc = mdbx_env_stat(env, &stat, sizeof(stat));
if (rc) {
error("mdbx_env_stat failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
lastpgno = info.me_last_pgno + 1;
errno = 0;
if (verbose) {
double k = 1024.0;
const char sf[] =
"KMGTPEZY"; /* LY: Kilo, Mega, Giga, Tera, Peta, Exa, Zetta, Yotta! */
for (i = 0; sf[i + 1] && info.me_mapsize / k > 1000.0; ++i)
k *= 1024;
print(" - map size %zu (%.2f %cb)\n", info.me_mapsize, info.me_mapsize / k,
sf[i]);
if (info.me_mapaddr)
print(" - mapaddr %p\n", info.me_mapaddr);
print(" - pagesize %u, max keysize %zu (%s), max readers %u\n",
stat.ms_psize, maxkeysize,
(maxkeysize == 511) ? "default" : (maxkeysize == 0) ? "devel"
: "custom",
info.me_maxreaders);
print(" - transactions: last %zu, bottom %zu, lag reading %zi\n",
info.me_last_txnid, info.me_tail_txnid,
info.me_last_txnid - info.me_tail_txnid);
print(" - meta-1: %s %zu, %s", meta_synctype(info.me_meta1_sign),
info.me_meta1_txnid, meta_lt(info.me_meta1_txnid, info.me_meta1_sign,
info.me_meta2_txnid, info.me_meta2_sign)
? "tail"
: "head");
if (info.me_meta1_txnid > info.me_last_txnid)
print(", rolled-back %zu (%zu >>> %zu)",
info.me_meta1_txnid - info.me_last_txnid, info.me_meta1_txnid,
info.me_last_txnid);
print("\n");
print(" - meta-2: %s %zu, %s", meta_synctype(info.me_meta2_sign),
info.me_meta2_txnid, meta_lt(info.me_meta2_txnid, info.me_meta2_sign,
info.me_meta1_txnid, info.me_meta1_sign)
? "tail"
: "head");
if (info.me_meta2_txnid > info.me_last_txnid)
print(", rolled-back %zu (%zu >>> %zu)",
info.me_meta2_txnid - info.me_last_txnid, info.me_meta2_txnid,
info.me_last_txnid);
print("\n");
}
if (exclusive > 1) {
if (verbose)
print(" - perform full check last-txn-id with meta-pages\n");
if (!meta_lt(info.me_meta1_txnid, info.me_meta1_sign, info.me_meta2_txnid,
info.me_meta2_sign) &&
info.me_meta1_txnid != info.me_last_txnid) {
print(" - meta-1 txn-id mismatch last-txn-id (%zi != %zi)\n",
info.me_meta1_txnid, info.me_last_txnid);
++problems_meta;
}
if (!meta_lt(info.me_meta2_txnid, info.me_meta2_sign, info.me_meta1_txnid,
info.me_meta1_sign) &&
info.me_meta2_txnid != info.me_last_txnid) {
print(" - meta-2 txn-id mismatch last-txn-id (%zi != %zi)\n",
info.me_meta2_txnid, info.me_last_txnid);
++problems_meta;
}
} else if (locktxn) {
if (verbose)
print(" - perform lite check last-txn-id with meta-pages (not a "
"monopolistic mode)\n");
size_t last = (info.me_meta2_txnid > info.me_meta1_txnid)
? info.me_meta2_txnid
: info.me_meta1_txnid;
if (last != info.me_last_txnid) {
print(" - last-meta mismatch last-txn-id (%zi != %zi)\n", last,
info.me_last_txnid);
++problems_meta;
}
} else if (verbose) {
print(" - skip check last-txn-id with meta-pages (monopolistic or "
"write-lock mode only)\n");
}
if (!dont_traversal) {
struct problem *saved_list;
size_t traversal_problems;
size_t empty_pages, lost_bytes;
print("Traversal b-tree...\n");
fflush(NULL);
walk.pagemap = calloc(lastpgno, sizeof(*walk.pagemap));
if (!walk.pagemap) {
rc = errno ? errno : ENOMEM;
error("calloc failed, error %d %s\n", rc, mdbx_strerror(rc));
goto bailout;
}
saved_list = problems_push();
rc = mdbx_env_pgwalk(txn, pgvisitor, NULL);
traversal_problems = problems_pop(saved_list);
if (rc) {
if (rc == EINTR && gotsignal) {
print(" - interrupted by signal\n");
fflush(NULL);
} else {
error("mdbx_env_pgwalk failed, error %d %s\n", rc, mdbx_strerror(rc));
}
goto bailout;
}
for (n = 0; n < lastpgno; ++n)
if (!walk.pagemap[n])
walk.dbi_pages[0] += 1;
empty_pages = lost_bytes = 0;
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
empty_pages += walk.dbi_empty_pages[i];
lost_bytes += walk.dbi_lost_bytes[i];
}
if (verbose) {
size_t total_page_bytes = walk.pgcount * stat.ms_psize;
print(" - dbi pages: %zu total", walk.pgcount);
if (verbose > 1)
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i)
print(", %s %zu", walk.dbi_names[i], walk.dbi_pages[i]);
print(", %s %zu\n", walk.dbi_names[0], walk.dbi_pages[0]);
if (verbose > 1) {
print(" - space info: total %zu bytes, payload %zu (%.1f%%), unused "
"%zu (%.1f%%)\n",
total_page_bytes, walk.total_payload_bytes,
walk.total_payload_bytes * 100.0 / total_page_bytes,
total_page_bytes - walk.total_payload_bytes,
(total_page_bytes - walk.total_payload_bytes) * 100.0 /
total_page_bytes);
for (i = 1; i < MAX_DBI && walk.dbi_names[i]; ++i) {
size_t dbi_bytes = walk.dbi_pages[i] * stat.ms_psize;
print(" %s: subtotal %zu bytes (%.1f%%), payload %zu (%.1f%%), "
"unused %zu (%.1f%%)",
walk.dbi_names[i], dbi_bytes,
dbi_bytes * 100.0 / total_page_bytes, walk.dbi_payload_bytes[i],
walk.dbi_payload_bytes[i] * 100.0 / dbi_bytes,
dbi_bytes - walk.dbi_payload_bytes[i],
(dbi_bytes - walk.dbi_payload_bytes[i]) * 100.0 / dbi_bytes);
if (walk.dbi_empty_pages[i])
print(", %zu empty pages", walk.dbi_empty_pages[i]);
if (walk.dbi_lost_bytes[i])
print(", %zu bytes lost", walk.dbi_lost_bytes[i]);
print("\n");
}
}
print(" - summary: average fill %.1f%%",
walk.total_payload_bytes * 100.0 / total_page_bytes);
if (empty_pages)
print(", %zu empty pages", empty_pages);
if (lost_bytes)
print(", %zu bytes lost", lost_bytes);
print(", %zu problems\n", traversal_problems);
}
} else if (verbose) {
print("Skipping b-tree walk...\n");
fflush(NULL);
}
if (!verbose)
print("Iterating DBIs...\n");
problems_maindb = process_db(-1, /* MAIN_DBI */ NULL, NULL, 0);
problems_freedb = process_db(0 /* FREE_DBI */, "free", handle_freedb, 0);
if (verbose) {
size_t value = info.me_mapsize / stat.ms_psize;
double percent = value / 100.0;
print(" - pages info: %zu total", value);
print(", allocated %zu (%.1f%%)", lastpgno, lastpgno / percent);
if (verbose > 1) {
value = info.me_mapsize / stat.ms_psize - lastpgno;
print(", remained %zu (%.1f%%)", value, value / percent);
value = lastpgno - freedb_pages;
print(", used %zu (%.1f%%)", value, value / percent);
print(", gc %zu (%.1f%%)", freedb_pages, freedb_pages / percent);
value = freedb_pages - reclaimable_pages;
print(", detained %zu (%.1f%%)", value, value / percent);
print(", reclaimable %zu (%.1f%%)", reclaimable_pages,
reclaimable_pages / percent);
}
value = info.me_mapsize / stat.ms_psize - lastpgno + reclaimable_pages;
print(", available %zu (%.1f%%)\n", value, value / percent);
}
if (problems_maindb == 0 && problems_freedb == 0) {
if (!dont_traversal && (exclusive || locktxn)) {
if (walk.pgcount != lastpgno - freedb_pages) {
error("used pages mismatch (%zu != %zu)\n", walk.pgcount,
lastpgno - freedb_pages);
}
if (walk.dbi_pages[0] != freedb_pages) {
error("gc pages mismatch (%zu != %zu)\n", walk.dbi_pages[0],
freedb_pages);
}
} else if (verbose) {
print(" - skip check used and gc pages (btree-traversal with "
"monopolistic or write-lock mode only)\n");
}
if (!process_db(-1, NULL, handle_maindb, 1)) {
if (!userdb_count && verbose)
print(" - does not contain multiple databases\n");
}
}
bailout:
if (txn)
mdbx_txn_abort(txn);
if (locktxn)
mdbx_txn_abort(locktxn);
if (env)
mdbx_env_close(env);
fflush(NULL);
if (rc) {
if (rc < 0)
return gotsignal ? EXIT_INTERRUPTED : EXIT_FAILURE_SYS;
return EXIT_FAILURE_MDB;
}
if (clock_gettime(CLOCK_MONOTONIC, &timestamp_finish)) {
rc = errno;
error("clock_gettime failed, error %d %s\n", rc, mdbx_strerror(rc));
return EXIT_FAILURE_SYS;
}
elapsed = timestamp_finish.tv_sec - timestamp_start.tv_sec +
(timestamp_finish.tv_nsec - timestamp_start.tv_nsec) * 1e-9;
total_problems += problems_meta;
if (total_problems || problems_maindb || problems_freedb) {
print("Total %zu error(s) is detected, elapsed %.3f seconds.\n",
total_problems, elapsed);
if (problems_meta || problems_maindb || problems_freedb)
return EXIT_FAILURE_CHECK_MAJOR;
return EXIT_FAILURE_CHECK_MINOR;
}
print("No error is detected, elapsed %.3f seconds\n", elapsed);
return EXIT_SUCCESS;
}

8
mdb_copy.1 → mdbx_copy.1
View File

@ -4,9 +4,9 @@
.\" Copying restrictions apply. See COPYRIGHT/LICENSE.
.TH MDB_COPY 1 "2014/06/20" "LMDB 0.9.14"
.SH NAME
mdb_copy \- LMDB environment copy tool
mdbx_copy \- LMDB environment copy tool
.SH SYNOPSIS
.B mdb_copy
.B mdbx_copy
[\c
.BR \-V ]
[\c
@ -18,7 +18,7 @@ mdb_copy \- LMDB environment copy tool
.BR dstpath ]
.SH DESCRIPTION
The
.B mdb_copy
.B mdbx_copy
utility copies an LMDB environment. The environment can
be copied regardless of whether it is currently in use.
No lockfile is created, since it gets recreated at need.
@ -52,6 +52,6 @@ This utility can trigger significant file size growth if run
in parallel with write transactions, because pages which they
free during copying cannot be reused until the copy is done.
.SH "SEE ALSO"
.BR mdb_stat (1)
.BR mdbx_stat (1)
.SH AUTHOR
Howard Chu of Symas Corporation <http://www.symas.com>

76
mdbx_copy.c Normal file
View File

@ -0,0 +1,76 @@
/* mdbx_copy.c - memory-mapped database backup tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2012-2017 Howard Chu, Symas Corp.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#include "mdbx.h"
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
static void sighandle(int sig) { (void)sig; }
int main(int argc, char *argv[]) {
int rc;
MDB_env *env = NULL;
const char *progname = argv[0], *act;
unsigned flags = MDB_RDONLY;
unsigned cpflags = 0;
for (; argc > 1 && argv[1][0] == '-'; argc--, argv++) {
if (argv[1][1] == 'n' && argv[1][2] == '\0')
flags |= MDB_NOSUBDIR;
else if (argv[1][1] == 'c' && argv[1][2] == '\0')
cpflags |= MDB_CP_COMPACT;
else if (argv[1][1] == 'V' && argv[1][2] == '\0') {
printf("%s\n", MDB_VERSION_STRING);
exit(0);
} else
argc = 0;
}
if (argc < 2 || argc > 3) {
fprintf(stderr, "usage: %s [-V] [-c] [-n] srcpath [dstpath]\n", progname);
exit(EXIT_FAILURE);
}
#ifdef SIGPIPE
signal(SIGPIPE, sighandle);
#endif
#ifdef SIGHUP
signal(SIGHUP, sighandle);
#endif
signal(SIGINT, sighandle);
signal(SIGTERM, sighandle);
act = "opening environment";
rc = mdbx_env_create(&env);
if (rc == MDB_SUCCESS) {
rc = mdbx_env_open(env, argv[1], flags, 0640);
}
if (rc == MDB_SUCCESS) {
act = "copying";
if (argc == 2)
rc = mdbx_env_copyfd2(env, STDOUT_FILENO, cpflags);
else
rc = mdbx_env_copy2(env, argv[2], cpflags);
}
if (rc)
fprintf(stderr, "%s: %s failed, error %d (%s)\n", progname, act, rc,
mdbx_strerror(rc));
mdbx_env_close(env);
return rc ? EXIT_FAILURE : EXIT_SUCCESS;
}

12
mdb_dump.1 → mdbx_dump.1
View File

@ -4,9 +4,9 @@
.\" Copying restrictions apply. See COPYRIGHT/LICENSE.
.TH MDB_DUMP 1 "2014/06/20" "LMDB 0.9.14"
.SH NAME
mdb_dump \- LMDB environment export tool
mdbx_dump \- LMDB environment export tool
.SH SYNOPSIS
.B mdb_dump
.B mdbx_dump
[\c
.BR \-V ]
[\c
@ -23,11 +23,11 @@ mdb_dump \- LMDB environment export tool
.BR \ envpath
.SH DESCRIPTION
The
.B mdb_dump
.B mdbx_dump
utility reads a database and writes its contents to the
standard output using a portable flat-text format
understood by the
.BR mdb_load (1)
.BR mdbx_load (1)
utility.
.SH OPTIONS
.TP
@ -69,9 +69,9 @@ will result in new databases that use the default comparison functions.
damaged beyond repair permitting neither record storage nor retrieval.\fP
The only available workaround is to modify the source for the
.BR mdb_load (1)
.BR mdbx_load (1)
utility to load the database using the correct comparison functions.
.SH "SEE ALSO"
.BR mdb_load (1)
.BR mdbx_load (1)
.SH AUTHOR
Howard Chu of Symas Corporation <http://www.symas.com>

316
mdbx_dump.c Normal file
View File

@ -0,0 +1,316 @@
/* mdbx_dump.c - memory-mapped database dump tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2011-2017 Howard Chu, Symas Corp.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#include "mdbx.h"
#include <ctype.h>
#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#define PRINT 1
static int mode;
typedef struct flagbit {
int bit;
char *name;
} flagbit;
flagbit dbflags[] = {{MDB_REVERSEKEY, "reversekey"},
{MDB_DUPSORT, "dupsort"},
{MDB_INTEGERKEY, "integerkey"},
{MDB_DUPFIXED, "dupfixed"},
{MDB_INTEGERDUP, "integerdup"},
{MDB_REVERSEDUP, "reversedup"},
{0, NULL}};
static volatile sig_atomic_t gotsig;
static void dumpsig(int sig) {
(void)sig;
gotsig = 1;
}
static const char hexc[] = "0123456789abcdef";
static void hex(unsigned char c) {
putchar(hexc[c >> 4]);
putchar(hexc[c & 0xf]);
}
static void text(MDB_val *v) {
unsigned char *c, *end;
putchar(' ');
c = v->mv_data;
end = c + v->mv_size;
while (c < end) {
if (isprint(*c)) {
putchar(*c);
} else {
putchar('\\');
hex(*c);
}
c++;
}
putchar('\n');
}
static void byte(MDB_val *v) {
unsigned char *c, *end;
putchar(' ');
c = v->mv_data;
end = c + v->mv_size;
while (c < end) {
hex(*c++);
}
putchar('\n');
}
/* Dump in BDB-compatible format */
static int dumpit(MDB_txn *txn, MDB_dbi dbi, char *name) {
MDB_cursor *mc;
MDBX_stat ms;
MDB_val key, data;
MDBX_envinfo info;
unsigned int flags;
int rc, i;
rc = mdbx_dbi_flags(txn, dbi, &flags);
if (rc)
return rc;
rc = mdbx_stat(txn, dbi, &ms, sizeof(ms));
if (rc)
return rc;
rc = mdbx_env_info(mdbx_txn_env(txn), &info, sizeof(info));
if (rc)
return rc;
printf("VERSION=3\n");
printf("format=%s\n", mode & PRINT ? "print" : "bytevalue");
if (name)
printf("database=%s\n", name);
printf("type=btree\n");
printf("mapsize=%zu\n", info.me_mapsize);
if (info.me_mapaddr)
printf("mapaddr=%p\n", info.me_mapaddr);
printf("maxreaders=%u\n", info.me_maxreaders);
for (i = 0; dbflags[i].bit; i++)
if (flags & dbflags[i].bit)
printf("%s=1\n", dbflags[i].name);
printf("db_pagesize=%d\n", ms.ms_psize);
printf("HEADER=END\n");
rc = mdbx_cursor_open(txn, dbi, &mc);
if (rc)
return rc;
while ((rc = mdbx_cursor_get(mc, &key, &data, MDB_NEXT)) == MDB_SUCCESS) {
if (gotsig) {
rc = EINTR;
break;
}
if (mode & PRINT) {
text(&key);
text(&data);
} else {
byte(&key);
byte(&data);
}
}
printf("DATA=END\n");
if (rc == MDB_NOTFOUND)
rc = MDB_SUCCESS;
return rc;
}
static void usage(char *prog) {
fprintf(stderr,
"usage: %s [-V] [-f output] [-l] [-n] [-p] [-a|-s subdb] dbpath\n",
prog);
exit(EXIT_FAILURE);
}
int main(int argc, char *argv[]) {
int i, rc;
MDB_env *env;
MDB_txn *txn;
MDB_dbi dbi;
char *prog = argv[0];
char *envname;
char *subname = NULL;
int alldbs = 0, envflags = 0, list = 0;
if (argc < 2) {
usage(prog);
}
/* -a: dump main DB and all subDBs
* -s: dump only the named subDB
* -n: use NOSUBDIR flag on env_open
* -p: use printable characters
* -f: write to file instead of stdout
* -V: print version and exit
* (default) dump only the main DB
*/
while ((i = getopt(argc, argv, "af:lnps:V")) != EOF) {
switch (i) {
case 'V':
printf("%s\n", MDB_VERSION_STRING);
exit(0);
break;
case 'l':
list = 1;
/*FALLTHROUGH*/;
case 'a':
if (subname)
usage(prog);
alldbs++;
break;
case 'f':
if (freopen(optarg, "w", stdout) == NULL) {
fprintf(stderr, "%s: %s: reopen: %s\n", prog, optarg, strerror(errno));
exit(EXIT_FAILURE);
}
break;
case 'n':
envflags |= MDB_NOSUBDIR;
break;
case 'p':
mode |= PRINT;
break;
case 's':
if (alldbs)
usage(prog);
subname = optarg;
break;
default:
usage(prog);
}
}
if (optind != argc - 1)
usage(prog);
#ifdef SIGPIPE
signal(SIGPIPE, dumpsig);
#endif
#ifdef SIGHUP
signal(SIGHUP, dumpsig);
#endif
signal(SIGINT, dumpsig);
signal(SIGTERM, dumpsig);
envname = argv[optind];
rc = mdbx_env_create(&env);
if (rc) {
fprintf(stderr, "mdbx_env_create failed, error %d %s\n", rc,
mdbx_strerror(rc));
return EXIT_FAILURE;
}
if (alldbs || subname) {
mdbx_env_set_maxdbs(env, 2);
}
rc = mdbx_env_open(env, envname, envflags | MDB_RDONLY, 0664);
if (rc) {
fprintf(stderr, "mdbx_env_open failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto env_close;
}
rc = mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn);
if (rc) {
fprintf(stderr, "mdbx_txn_begin failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto env_close;
}
rc = mdbx_dbi_open(txn, subname, 0, &dbi);
if (rc) {
fprintf(stderr, "mdbx_open failed, error %d %s\n", rc, mdbx_strerror(rc));
goto txn_abort;
}
if (alldbs) {
MDB_cursor *cursor;
MDB_val key;
int count = 0;
rc = mdbx_cursor_open(txn, dbi, &cursor);
if (rc) {
fprintf(stderr, "mdbx_cursor_open failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto txn_abort;
}
while ((rc = mdbx_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP)) == 0) {
char *str;
MDB_dbi db2;
if (memchr(key.mv_data, '\0', key.mv_size))
continue;
count++;
str = malloc(key.mv_size + 1);
memcpy(str, key.mv_data, key.mv_size);
str[key.mv_size] = '\0';
rc = mdbx_dbi_open(txn, str, 0, &db2);
if (rc == MDB_SUCCESS) {
if (list) {
printf("%s\n", str);
list++;
} else {
rc = dumpit(txn, db2, str);
if (rc)
break;
}
mdbx_dbi_close(env, db2);
}
free(str);
if (rc)
continue;
}
mdbx_cursor_close(cursor);
if (!count) {
fprintf(stderr, "%s: %s does not contain multiple databases\n", prog,
envname);
rc = MDB_NOTFOUND;
} else if (rc == MDB_INCOMPATIBLE) {
/* LY: the record it not a named sub-db. */
rc = MDB_SUCCESS;
}
} else {
rc = dumpit(txn, dbi, subname);
}
if (rc && rc != MDB_NOTFOUND)
fprintf(stderr, "%s: %s: %s\n", prog, envname, mdbx_strerror(rc));
mdbx_dbi_close(env, dbi);
txn_abort:
mdbx_txn_abort(txn);
env_close:
mdbx_env_close(env);
return rc ? EXIT_FAILURE : EXIT_SUCCESS;
}

14
mdb_load.1 → mdbx_load.1
View File

@ -4,9 +4,9 @@
.\" Copying restrictions apply. See COPYRIGHT/LICENSE.
.TH MDB_LOAD 1 "2014/06/20" "LMDB 0.9.14"
.SH NAME
mdb_load \- LMDB environment import tool
mdbx_load \- LMDB environment import tool
.SH SYNOPSIS
.B mdb_load
.B mdbx_load
[\c
.BR \-V ]
[\c
@ -22,15 +22,15 @@ mdb_load \- LMDB environment import tool
.BR \ envpath
.SH DESCRIPTION
The
.B mdb_load
.B mdbx_load
utility reads from the standard input and loads it into the
LMDB environment
.BR envpath .
The input to
.B mdb_load
.B mdbx_load
must be in the output format specified by the
.BR mdb_dump (1)
.BR mdbx_dump (1)
utility or as specified by the
.B -T
option below.
@ -66,7 +66,7 @@ character; for example, \\0a is a newline character in the ASCII character set.
For this reason, any backslash or newline characters that naturally occur in the text
input must be escaped to avoid misinterpretation by
.BR mdb_load .
.BR mdbx_load .
.SH DIAGNOSTICS
Exit status is zero if no errors occur.
@ -74,6 +74,6 @@ Errors result in a non-zero exit status and
a diagnostic message being written to standard error.
.SH "SEE ALSO"
.BR mdb_dump (1)
.BR mdbx_dump (1)
.SH AUTHOR
Howard Chu of Symas Corporation <http://www.symas.com>

466
mdbx_load.c Normal file
View File

@ -0,0 +1,466 @@
/* mdbx_load.c - memory-mapped database load tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2011-2017 Howard Chu, Symas Corp.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#include "mdbx.h"
#include <ctype.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#define PRINT 1
#define NOHDR 2
static int mode;
static char *subname = NULL;
static size_t lineno;
static int version;
static int dbi_flags;
static char *prog;
static int Eof;
static MDBX_envinfo info;
static MDB_val kbuf, dbuf;
#define STRLENOF(s) (sizeof(s) - 1)
typedef struct flagbit {
int bit;
char *name;
int len;
} flagbit;
#define S(s) s, STRLENOF(s)
flagbit dbflags[] = {{MDB_REVERSEKEY, S("reversekey")},
{MDB_DUPSORT, S("dupsort")},
{MDB_INTEGERKEY, S("integerkey")},
{MDB_DUPFIXED, S("dupfixed")},
{MDB_INTEGERDUP, S("integerdup")},
{MDB_REVERSEDUP, S("reversedup")},
{0, NULL, 0}};
static void readhdr(void) {
char *ptr;
dbi_flags = 0;
while (fgets(dbuf.mv_data, dbuf.mv_size, stdin) != NULL) {
lineno++;
if (!strncmp(dbuf.mv_data, "db_pagesize=", STRLENOF("db_pagesize=")) ||
!strncmp(dbuf.mv_data, "duplicates=", STRLENOF("duplicates="))) {
/* LY: silently ignore information fields. */
continue;
} else if (!strncmp(dbuf.mv_data, "VERSION=", STRLENOF("VERSION="))) {
version = atoi((char *)dbuf.mv_data + STRLENOF("VERSION="));
if (version > 3) {
fprintf(stderr, "%s: line %zd: unsupported VERSION %d\n", prog, lineno,
version);
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "HEADER=END", STRLENOF("HEADER=END"))) {
break;
} else if (!strncmp(dbuf.mv_data, "format=", STRLENOF("format="))) {
if (!strncmp((char *)dbuf.mv_data + STRLENOF("FORMAT="), "print",
STRLENOF("print")))
mode |= PRINT;
else if (strncmp((char *)dbuf.mv_data + STRLENOF("FORMAT="), "bytevalue",
STRLENOF("bytevalue"))) {
fprintf(stderr, "%s: line %zd: unsupported FORMAT %s\n", prog, lineno,
(char *)dbuf.mv_data + STRLENOF("FORMAT="));
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "database=", STRLENOF("database="))) {
ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
if (ptr)
*ptr = '\0';
if (subname)
free(subname);
subname = strdup((char *)dbuf.mv_data + STRLENOF("database="));
} else if (!strncmp(dbuf.mv_data, "type=", STRLENOF("type="))) {
if (strncmp((char *)dbuf.mv_data + STRLENOF("type="), "btree",
STRLENOF("btree"))) {
fprintf(stderr, "%s: line %zd: unsupported type %s\n", prog, lineno,
(char *)dbuf.mv_data + STRLENOF("type="));
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "mapaddr=", STRLENOF("mapaddr="))) {
int i;
ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
if (ptr)
*ptr = '\0';
i = sscanf((char *)dbuf.mv_data + STRLENOF("mapaddr="), "%p",
&info.me_mapaddr);
if (i != 1) {
fprintf(stderr, "%s: line %zd: invalid mapaddr %s\n", prog, lineno,
(char *)dbuf.mv_data + STRLENOF("mapaddr="));
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "mapsize=", STRLENOF("mapsize="))) {
int i;
ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
if (ptr)
*ptr = '\0';
i = sscanf((char *)dbuf.mv_data + STRLENOF("mapsize="), "%zu",
&info.me_mapsize);
if (i != 1) {
fprintf(stderr, "%s: line %zd: invalid mapsize %s\n", prog, lineno,
(char *)dbuf.mv_data + STRLENOF("mapsize="));
exit(EXIT_FAILURE);
}
} else if (!strncmp(dbuf.mv_data, "maxreaders=", STRLENOF("maxreaders="))) {
int i;
ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
if (ptr)
*ptr = '\0';
i = sscanf((char *)dbuf.mv_data + STRLENOF("maxreaders="), "%u",
&info.me_maxreaders);
if (i != 1) {
fprintf(stderr, "%s: line %zd: invalid maxreaders %s\n", prog, lineno,
(char *)dbuf.mv_data + STRLENOF("maxreaders="));
exit(EXIT_FAILURE);
}
} else {
int i;
for (i = 0; dbflags[i].bit; i++) {
if (!strncmp(dbuf.mv_data, dbflags[i].name, dbflags[i].len) &&
((char *)dbuf.mv_data)[dbflags[i].len] == '=') {
if (((char *)dbuf.mv_data)[dbflags[i].len + 1] == '1')
dbi_flags |= dbflags[i].bit;
break;
}
}
if (!dbflags[i].bit) {
ptr = memchr(dbuf.mv_data, '=', dbuf.mv_size);
if (!ptr) {
fprintf(stderr, "%s: line %zd: unexpected format\n", prog, lineno);
exit(EXIT_FAILURE);
} else {
*ptr = '\0';
fprintf(stderr, "%s: line %zd: unrecognized keyword ignored: %s\n",
prog, lineno, (char *)dbuf.mv_data);
}
}
}
}
}
static void badend(void) {
fprintf(stderr, "%s: line %zd: unexpected end of input\n", prog, lineno);
}
static int unhex(unsigned char *c2) {
int x, c;
x = *c2++ & 0x4f;
if (x & 0x40)
x -= 55;
c = x << 4;
x = *c2 & 0x4f;
if (x & 0x40)
x -= 55;
c |= x;
return c;
}
static int readline(MDB_val *out, MDB_val *buf) {
unsigned char *c1, *c2, *end;
size_t len, l2;
int c;
if (!(mode & NOHDR)) {
c = fgetc(stdin);
if (c == EOF) {
Eof = 1;
return EOF;
}
if (c != ' ') {
lineno++;
if (fgets(buf->mv_data, buf->mv_size, stdin) == NULL) {
badend:
Eof = 1;
badend();
return EOF;
}
if (c == 'D' && !strncmp(buf->mv_data, "ATA=END", STRLENOF("ATA=END")))
return EOF;
goto badend;
}
}
if (fgets(buf->mv_data, buf->mv_size, stdin) == NULL) {
Eof = 1;
return EOF;
}
lineno++;
c1 = buf->mv_data;
len = strlen((char *)c1);
l2 = len;
/* Is buffer too short? */
while (c1[len - 1] != '\n') {
buf->mv_data = realloc(buf->mv_data, buf->mv_size * 2);
if (!buf->mv_data) {
Eof = 1;
fprintf(stderr, "%s: line %zd: out of memory, line too long\n", prog,
lineno);
return EOF;
}
c1 = buf->mv_data;
c1 += l2;
if (fgets((char *)c1, buf->mv_size + 1, stdin) == NULL) {
Eof = 1;
badend();
return EOF;
}
buf->mv_size *= 2;
len = strlen((char *)c1);
l2 += len;
}
c1 = c2 = buf->mv_data;
len = l2;
c1[--len] = '\0';
end = c1 + len;
if (mode & PRINT) {
while (c2 < end) {
if (*c2 == '\\') {
if (c2[1] == '\\') {
c1++;
c2 += 2;
} else {
if (c2 + 3 > end || !isxdigit(c2[1]) || !isxdigit(c2[2])) {
Eof = 1;
badend();
return EOF;
}
*c1++ = unhex(++c2);
c2 += 2;
}
} else {
/* copies are redundant when no escapes were used */
*c1++ = *c2++;
}
}
} else {
/* odd length not allowed */
if (len & 1) {
Eof = 1;
badend();
return EOF;
}
while (c2 < end) {
if (!isxdigit(*c2) || !isxdigit(c2[1])) {
Eof = 1;
badend();
return EOF;
}
*c1++ = unhex(c2);
c2 += 2;
}
}
c2 = out->mv_data = buf->mv_data;
out->mv_size = c1 - c2;
return 0;
}
static void usage(void) {
fprintf(stderr, "usage: %s [-V] [-f input] [-n] [-s name] [-N] [-T] dbpath\n",
prog);
exit(EXIT_FAILURE);
}
int main(int argc, char *argv[]) {
int i, rc;
MDB_env *env;
MDB_txn *txn;
MDB_cursor *mc;
MDB_dbi dbi;
char *envname;
int envflags = 0, putflags = 0;
prog = argv[0];
if (argc < 2) {
usage();
}
/* -f: load file instead of stdin
* -n: use NOSUBDIR flag on env_open
* -s: load into named subDB
* -N: use NOOVERWRITE on puts
* -T: read plaintext
* -V: print version and exit
*/
while ((i = getopt(argc, argv, "f:ns:NTV")) != EOF) {
switch (i) {
case 'V':
printf("%s\n", MDB_VERSION_STRING);
exit(0);
break;
case 'f':
if (freopen(optarg, "r", stdin) == NULL) {
fprintf(stderr, "%s: %s: reopen: %s\n", prog, optarg, strerror(errno));
exit(EXIT_FAILURE);
}
break;
case 'n':
envflags |= MDB_NOSUBDIR;
break;
case 's':
subname = strdup(optarg);
break;
case 'N':
putflags = MDB_NOOVERWRITE | MDB_NODUPDATA;
break;
case 'T':
mode |= NOHDR | PRINT;
break;
default:
usage();
}
}
if (optind != argc - 1)
usage();
dbuf.mv_size = 4096;
dbuf.mv_data = malloc(dbuf.mv_size);
if (!(mode & NOHDR))
readhdr();
envname = argv[optind];
rc = mdbx_env_create(&env);
if (rc) {
fprintf(stderr, "mdbx_env_create failed, error %d %s\n", rc,
mdbx_strerror(rc));
return EXIT_FAILURE;
}
mdbx_env_set_maxdbs(env, 2);
if (info.me_maxreaders)
mdbx_env_set_maxreaders(env, info.me_maxreaders);
if (info.me_mapsize)
mdbx_env_set_mapsize(env, info.me_mapsize);
if (info.me_mapaddr)
envflags |= MDB_FIXEDMAP;
rc = mdbx_env_open(env, envname, envflags, 0664);
if (rc) {
fprintf(stderr, "mdbx_env_open failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto env_close;
}
kbuf.mv_size = mdbx_env_get_maxkeysize(env) * 2 + 2;
kbuf.mv_data = malloc(kbuf.mv_size);
while (!Eof) {
MDB_val key, data;
int batch = 0;
rc = mdbx_txn_begin(env, NULL, 0, &txn);
if (rc) {
fprintf(stderr, "mdbx_txn_begin failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto env_close;
}
rc = mdbx_dbi_open(txn, subname, dbi_flags | MDB_CREATE, &dbi);
if (rc) {
fprintf(stderr, "mdbx_open failed, error %d %s\n", rc, mdbx_strerror(rc));
goto txn_abort;
}
rc = mdbx_cursor_open(txn, dbi, &mc);
if (rc) {
fprintf(stderr, "mdbx_cursor_open failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto txn_abort;
}
while (1) {
rc = readline(&key, &kbuf);
if (rc) /* rc == EOF */
break;
rc = readline(&data, &dbuf);
if (rc) {
fprintf(stderr, "%s: line %zd: failed to read key value\n", prog,
lineno);
goto txn_abort;
}
rc = mdbx_cursor_put(mc, &key, &data, putflags);
if (rc == MDB_KEYEXIST && putflags)
continue;
if (rc) {
fprintf(stderr, "mdbx_cursor_put failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto txn_abort;
}
batch++;
if (batch == 100) {
rc = mdbx_txn_commit(txn);
if (rc) {
fprintf(stderr, "%s: line %zd: txn_commit: %s\n", prog, lineno,
mdbx_strerror(rc));
goto env_close;
}
rc = mdbx_txn_begin(env, NULL, 0, &txn);
if (rc) {
fprintf(stderr, "mdbx_txn_begin failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto env_close;
}
rc = mdbx_cursor_open(txn, dbi, &mc);
if (rc) {
fprintf(stderr, "mdbx_cursor_open failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto txn_abort;
}
batch = 0;
}
}
rc = mdbx_txn_commit(txn);
txn = NULL;
if (rc) {
fprintf(stderr, "%s: line %zd: txn_commit: %s\n", prog, lineno,
mdbx_strerror(rc));
goto env_close;
}
mdbx_dbi_close(env, dbi);
if (!(mode & NOHDR))
readhdr();
}
txn_abort:
mdbx_txn_abort(txn);
env_close:
mdbx_env_close(env);
return rc ? EXIT_FAILURE : EXIT_SUCCESS;
}

8
mdb_stat.1 → mdbx_stat.1
View File

@ -4,9 +4,9 @@
.\" Copying restrictions apply. See COPYRIGHT/LICENSE.
.TH MDB_STAT 1 "2014/06/20" "LMDB 0.9.14"
.SH NAME
mdb_stat \- LMDB environment status tool
mdbx_stat \- LMDB environment status tool
.SH SYNOPSIS
.B mdb_stat
.B mdbx_stat
[\c
.BR \-V ]
[\c
@ -23,7 +23,7 @@ mdb_stat \- LMDB environment status tool
.BR \ envpath
.SH DESCRIPTION
The
.B mdb_stat
.B mdbx_stat
utility displays the status of an LMDB environment.
.SH OPTIONS
.TP
@ -61,6 +61,6 @@ Exit status is zero if no errors occur.
Errors result in a non-zero exit status and
a diagnostic message being written to standard error.
.SH "SEE ALSO"
.BR mdb_copy (1)
.BR mdbx_copy (1)
.SH AUTHOR
Howard Chu of Symas Corporation <http://www.symas.com>

306
mdbx_stat.c Normal file
View File

@ -0,0 +1,306 @@
/* mdbx_stat.c - memory-mapped database status tool */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2011-2017 Howard Chu, Symas Corp.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#include "mdbx.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
static void prstat(MDBX_stat *ms) {
printf(" Page size: %u\n", ms->ms_psize);
printf(" Tree depth: %u\n", ms->ms_depth);
printf(" Branch pages: %zu\n", ms->ms_branch_pages);
printf(" Leaf pages: %zu\n", ms->ms_leaf_pages);
printf(" Overflow pages: %zu\n", ms->ms_overflow_pages);
printf(" Entries: %zu\n", ms->ms_entries);
}
static void usage(char *prog) {
fprintf(stderr,
"usage: %s [-V] [-n] [-e] [-r[r]] [-f[f[f]]] [-a|-s subdb] dbpath\n",
prog);
exit(EXIT_FAILURE);
}
int main(int argc, char *argv[]) {
int i, rc;
MDB_env *env;
MDB_txn *txn;
MDB_dbi dbi;
MDBX_stat mst;
MDBX_envinfo mei;
char *prog = argv[0];
char *envname;
char *subname = NULL;
int alldbs = 0, envinfo = 0, envflags = 0, freinfo = 0, rdrinfo = 0;
if (argc < 2) {
usage(prog);
}
/* -a: print stat of main DB and all subDBs
* -s: print stat of only the named subDB
* -e: print env info
* -f: print freelist info
* -r: print reader info
* -n: use NOSUBDIR flag on env_open
* -V: print version and exit
* (default) print stat of only the main DB
*/
while ((i = getopt(argc, argv, "Vaefnrs:")) != EOF) {
switch (i) {
case 'V':
printf("%s\n", MDB_VERSION_STRING);
exit(0);
break;
case 'a':
if (subname)
usage(prog);
alldbs++;
break;
case 'e':
envinfo++;
break;
case 'f':
freinfo++;
break;
case 'n':
envflags |= MDB_NOSUBDIR;
break;
case 'r':
rdrinfo++;
break;
case 's':
if (alldbs)
usage(prog);
subname = optarg;
break;
default:
usage(prog);
}
}
if (optind != argc - 1)
usage(prog);
envname = argv[optind];
rc = mdbx_env_create(&env);
if (rc) {
fprintf(stderr, "mdbx_env_create failed, error %d %s\n", rc,
mdbx_strerror(rc));
return EXIT_FAILURE;
}
if (alldbs || subname) {
mdbx_env_set_maxdbs(env, 4);
}
rc = mdbx_env_open(env, envname, envflags | MDB_RDONLY, 0664);
if (rc) {
fprintf(stderr, "mdbx_env_open failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto env_close;
}
if (envinfo) {
(void)mdbx_env_stat(env, &mst, sizeof(mst));
(void)mdbx_env_info(env, &mei, sizeof(mei));
printf("Environment Info\n");
printf(" Map address: %p\n", mei.me_mapaddr);
printf(" Map size: %zu\n", mei.me_mapsize);
printf(" Page size: %u\n", mst.ms_psize);
printf(" Max pages: %zu\n", mei.me_mapsize / mst.ms_psize);
printf(" Number of pages used: %zu\n", mei.me_last_pgno + 1);
printf(" Last transaction ID: %zu\n", mei.me_last_txnid);
printf(" Tail transaction ID: %zu (%zi)\n", mei.me_tail_txnid,
mei.me_tail_txnid - mei.me_last_txnid);
printf(" Max readers: %u\n", mei.me_maxreaders);
printf(" Number of readers used: %u\n", mei.me_numreaders);
} else {
/* LY: zap warnings from gcc */
memset(&mst, 0, sizeof(mst));
memset(&mei, 0, sizeof(mei));
}
if (rdrinfo) {
printf("Reader Table Status\n");
rc = mdbx_reader_list(env, (MDB_msg_func *)fputs, stdout);
if (rdrinfo > 1) {
int dead;
mdbx_reader_check(env, &dead);
printf(" %d stale readers cleared.\n", dead);
rc = mdbx_reader_list(env, (MDB_msg_func *)fputs, stdout);
}
if (!(subname || alldbs || freinfo))
goto env_close;
}
rc = mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn);
if (rc) {
fprintf(stderr, "mdbx_txn_begin failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto env_close;
}
if (freinfo) {
MDB_cursor *cursor;
MDB_val key, data;
size_t pages = 0, *iptr;
size_t reclaimable = 0;
printf("Freelist Status\n");
dbi = 0;
rc = mdbx_cursor_open(txn, dbi, &cursor);
if (rc) {
fprintf(stderr, "mdbx_cursor_open failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto txn_abort;
}
rc = mdbx_stat(txn, dbi, &mst, sizeof(mst));
if (rc) {
fprintf(stderr, "mdbx_stat failed, error %d %s\n", rc, mdbx_strerror(rc));
goto txn_abort;
}
prstat(&mst);
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
iptr = data.mv_data;
pages += *iptr;
if (envinfo && mei.me_tail_txnid > *(size_t *)key.mv_data)
reclaimable += *iptr;
if (freinfo > 1) {
char *bad = "";
size_t pg, prev;
ssize_t i, j, span = 0;
j = *iptr++;
for (i = j, prev = 1; --i >= 0;) {
pg = iptr[i];
if (pg <= prev)
bad = " [bad sequence]";
prev = pg;
pg += span;
for (; i >= span && iptr[i - span] == pg; span++, pg++)
;
}
printf(" Transaction %zu, %zd pages, maxspan %zd%s\n",
*(size_t *)key.mv_data, j, span, bad);
if (freinfo > 2) {
for (--j; j >= 0;) {
pg = iptr[j];
for (span = 1; --j >= 0 && iptr[j] == pg + span; span++)
;
if (span > 1)
printf(" %9zu[%zd]\n", pg, span);
else
printf(" %9zu\n", pg);
}
}
}
}
mdbx_cursor_close(cursor);
if (envinfo) {
size_t value = mei.me_mapsize / mst.ms_psize;
double percent = value / 100.0;
printf("Page Allocation Info\n");
printf(" Max pages: %9zu 100%%\n", value);
value = mei.me_last_pgno + 1;
printf(" Number of pages used: %zu %.1f%%\n", value, value / percent);
value = mei.me_mapsize / mst.ms_psize - (mei.me_last_pgno + 1);
printf(" Remained: %zu %.1f%%\n", value, value / percent);
value = mei.me_last_pgno + 1 - pages;
printf(" Used now: %zu %.1f%%\n", value, value / percent);
value = pages;
printf(" Unallocated: %zu %.1f%%\n", value, value / percent);
value = pages - reclaimable;
printf(" Detained: %zu %.1f%%\n", value, value / percent);
value = reclaimable;
printf(" Reclaimable: %zu %.1f%%\n", value, value / percent);
value =
mei.me_mapsize / mst.ms_psize - (mei.me_last_pgno + 1) + reclaimable;
printf(" Available: %zu %.1f%%\n", value, value / percent);
} else
printf(" Free pages: %zu\n", pages);
}
rc = mdbx_dbi_open(txn, subname, 0, &dbi);
if (rc) {
fprintf(stderr, "mdbx_open failed, error %d %s\n", rc, mdbx_strerror(rc));
goto txn_abort;
}
rc = mdbx_stat(txn, dbi, &mst, sizeof(mst));
if (rc) {
fprintf(stderr, "mdbx_stat failed, error %d %s\n", rc, mdbx_strerror(rc));
goto txn_abort;
}
printf("Status of %s\n", subname ? subname : "Main DB");
prstat(&mst);
if (alldbs) {
MDB_cursor *cursor;
MDB_val key;
rc = mdbx_cursor_open(txn, dbi, &cursor);
if (rc) {
fprintf(stderr, "mdbx_cursor_open failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto txn_abort;
}
while ((rc = mdbx_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP)) == 0) {
char *str;
MDB_dbi db2;
if (memchr(key.mv_data, '\0', key.mv_size))
continue;
str = malloc(key.mv_size + 1);
memcpy(str, key.mv_data, key.mv_size);
str[key.mv_size] = '\0';
rc = mdbx_dbi_open(txn, str, 0, &db2);
if (rc == MDB_SUCCESS)
printf("Status of %s\n", str);
free(str);
if (rc)
continue;
rc = mdbx_stat(txn, db2, &mst, sizeof(mst));
if (rc) {
fprintf(stderr, "mdbx_stat failed, error %d %s\n", rc,
mdbx_strerror(rc));
goto txn_abort;
}
prstat(&mst);
mdbx_dbi_close(env, db2);
}
mdbx_cursor_close(cursor);
}
if (rc == MDB_NOTFOUND)
rc = MDB_SUCCESS;
mdbx_dbi_close(env, dbi);
txn_abort:
mdbx_txn_abort(txn);
env_close:
mdbx_env_close(env);
return rc ? EXIT_FAILURE : EXIT_SUCCESS;
}

361
midl.c
View File

@ -1,361 +0,0 @@
/** @file midl.c
* @brief ldap bdb back-end ID List functions */
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2000-2017 The OpenLDAP Foundation.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#include <limits.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include "midl.h"
/** @defgroup internal LMDB Internals
* @{
*/
/** @defgroup idls ID List Management
* @{
*/
static unsigned __hot
mdb_midl_search( MDB_IDL ids, MDB_ID id )
{
/*
* binary search of id in ids
* if found, returns position of id
* if not found, returns first position greater than id
*/
unsigned base = 0;
unsigned cursor = 1;
int val = 0;
unsigned n = ids[0];
while( 0 < n ) {
unsigned pivot = n >> 1;
cursor = base + pivot + 1;
val = mdbx_cmp2int( ids[cursor], id );
if( val < 0 ) {
n = pivot;
} else if ( val > 0 ) {
base = cursor;
n -= pivot + 1;
} else {
return cursor;
}
}
if( val > 0 ) {
++cursor;
}
return cursor;
}
#if 0 /* superseded by append/sort */
static int mdb_midl_insert( MDB_IDL ids, MDB_ID id )
{
unsigned x, i;
x = mdb_midl_search( ids, id );
assert( x > 0 );
if( x < 1 ) {
/* internal error */
return -2;
}
if ( x <= ids[0] && ids[x] == id ) {
/* duplicate */
assert(0);
return -1;
}
if ( ++ids[0] >= MDB_IDL_DB_MAX ) {
/* no room */
--ids[0];
return -2;
} else {
/* insert id */
for (i=ids[0]; i>x; i--)
ids[i] = ids[i-1];
ids[x] = id;
}
return 0;
}
#endif
static MDB_IDL mdb_midl_alloc(int num)
{
MDB_IDL ids = malloc((num+2) * sizeof(MDB_ID));
if (ids) {
*ids++ = num;
*ids = 0;
}
return ids;
}
static void mdb_midl_free(MDB_IDL ids)
{
if (ids)
free(ids-1);
}
static void mdb_midl_shrink( MDB_IDL *idp )
{
MDB_IDL ids = *idp;
if (*(--ids) > MDB_IDL_UM_MAX &&
(ids = realloc(ids, (MDB_IDL_UM_MAX+2) * sizeof(MDB_ID))))
{
*ids++ = MDB_IDL_UM_MAX;
*idp = ids;
}
}
static int mdb_midl_grow( MDB_IDL *idp, int num )
{
MDB_IDL idn = *idp-1;
/* grow it */
idn = realloc(idn, (*idn + num + 2) * sizeof(MDB_ID));
if (!idn)
return ENOMEM;
*idn++ += num;
*idp = idn;
return 0;
}
static int mdb_midl_need( MDB_IDL *idp, unsigned num )
{
MDB_IDL ids = *idp;
num += ids[0];
if (num > ids[-1]) {
num = (num + num/4 + (256 + 2)) & -256;
if (!(ids = realloc(ids-1, num * sizeof(MDB_ID))))
return ENOMEM;
*ids++ = num - 2;
*idp = ids;
}
return 0;
}
static int mdb_midl_append( MDB_IDL *idp, MDB_ID id )
{
MDB_IDL ids = *idp;
/* Too big? */
if (ids[0] >= ids[-1]) {
if (mdb_midl_grow(idp, MDB_IDL_UM_MAX))
return ENOMEM;
ids = *idp;
}
ids[0]++;
ids[ids[0]] = id;
return 0;
}
static int mdb_midl_append_list( MDB_IDL *idp, MDB_IDL app )
{
MDB_IDL ids = *idp;
/* Too big? */
if (ids[0] + app[0] >= ids[-1]) {
if (mdb_midl_grow(idp, app[0]))
return ENOMEM;
ids = *idp;
}
memcpy(&ids[ids[0]+1], &app[1], app[0] * sizeof(MDB_ID));
ids[0] += app[0];
return 0;
}
static int mdb_midl_append_range( MDB_IDL *idp, MDB_ID id, unsigned n )
{
MDB_ID *ids = *idp, len = ids[0];
/* Too big? */
if (len + n > ids[-1]) {
if (mdb_midl_grow(idp, n | MDB_IDL_UM_MAX))
return ENOMEM;
ids = *idp;
}
ids[0] = len + n;
ids += len;
while (n)
ids[n--] = id++;
return 0;
}
static void __hot
mdb_midl_xmerge( MDB_IDL idl, MDB_IDL merge )
{
MDB_ID old_id, merge_id, i = merge[0], j = idl[0], k = i+j, total = k;
idl[0] = (MDB_ID)-1; /* delimiter for idl scan below */
old_id = idl[j];
while (i) {
merge_id = merge[i--];
for (; old_id < merge_id; old_id = idl[--j])
idl[k--] = old_id;
idl[k--] = merge_id;
}
idl[0] = total;
}
/* Quicksort + Insertion sort for small arrays */
#define SMALL 8
#define MIDL_SWAP(a,b) { itmp=(a); (a)=(b); (b)=itmp; }
static void __hot
mdb_midl_sort( MDB_IDL ids )
{
/* Max possible depth of int-indexed tree * 2 items/level */
int istack[sizeof(int)*CHAR_BIT * 2];
int i,j,k,l,ir,jstack;
MDB_ID a, itmp;
ir = (int)ids[0];
l = 1;
jstack = 0;
for(;;) {
if (ir - l < SMALL) { /* Insertion sort */
for (j=l+1;j<=ir;j++) {
a = ids[j];
for (i=j-1;i>=1;i--) {
if (ids[i] >= a) break;
ids[i+1] = ids[i];
}
ids[i+1] = a;
}
if (jstack == 0) break;
ir = istack[jstack--];
l = istack[jstack--];
} else {
k = (l + ir) >> 1; /* Choose median of left, center, right */
MIDL_SWAP(ids[k], ids[l+1]);
if (ids[l] < ids[ir]) {
MIDL_SWAP(ids[l], ids[ir]);
}
if (ids[l+1] < ids[ir]) {
MIDL_SWAP(ids[l+1], ids[ir]);
}
if (ids[l] < ids[l+1]) {
MIDL_SWAP(ids[l], ids[l+1]);
}
i = l+1;
j = ir;
a = ids[l+1];
for(;;) {
do i++; while(ids[i] > a);
do j--; while(ids[j] < a);
if (j < i) break;
MIDL_SWAP(ids[i],ids[j]);
}
ids[l+1] = ids[j];
ids[j] = a;
jstack += 2;
if (ir-i+1 >= j-l) {
istack[jstack] = ir;
istack[jstack-1] = i;
ir = j-1;
} else {
istack[jstack] = j-1;
istack[jstack-1] = l;
l = i;
}
}
}
}
static unsigned __hot
mdb_mid2l_search( MDB_ID2L ids, MDB_ID id )
{
/*
* binary search of id in ids
* if found, returns position of id
* if not found, returns first position greater than id
*/
unsigned base = 0;
unsigned cursor = 1;
int val = 0;
unsigned n = (unsigned)ids[0].mid;
while( 0 < n ) {
unsigned pivot = n >> 1;
cursor = base + pivot + 1;
val = mdbx_cmp2int( id, ids[cursor].mid );
if( val < 0 ) {
n = pivot;
} else if ( val > 0 ) {
base = cursor;
n -= pivot + 1;
} else {
return cursor;
}
}
if( val > 0 ) {
++cursor;
}
return cursor;
}
static int mdb_mid2l_insert( MDB_ID2L ids, MDB_ID2 *id )
{
unsigned x, i;
x = mdb_mid2l_search( ids, id->mid );
if( x < 1 ) {
/* internal error */
return -2;
}
if ( x <= ids[0].mid && ids[x].mid == id->mid ) {
/* duplicate */
return -1;
}
if ( ids[0].mid >= MDB_IDL_UM_MAX ) {
/* too big */
return -2;
} else {
/* insert id */
ids[0].mid++;
for (i=(unsigned)ids[0].mid; i>x; i--)
ids[i] = ids[i-1];
ids[x] = *id;
}
return 0;
}
static int mdb_mid2l_append( MDB_ID2L ids, MDB_ID2 *id )
{
/* Too big? */
if (ids[0].mid >= MDB_IDL_UM_MAX) {
return -2;
}
ids[0].mid++;
ids[ids[0].mid] = *id;
return 0;
}
/** @} */
/** @} */

147
midl.h
View File

@ -1,45 +1,3 @@
/** @file midl.h
* @brief LMDB ID List header file.
*
* This file was originally part of back-bdb but has been
* modified for use in libmdb. Most of the macros defined
* in this file are unused, just left over from the original.
*
* This file is only used internally in libmdb and its definitions
* are not exposed publicly.
*/
/*
* Copyright 2015-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2000-2017 The OpenLDAP Foundation.
* Copyright 2015,2016 Peter-Service R&D LLC.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted only as authorized by the OpenLDAP
* Public License.
*
* A copy of this license is available in the file LICENSE in the
* top-level directory of the distribution or, alternatively, at
* <http://www.OpenLDAP.org/license.html>.
*/
#ifndef _MDB_MIDL_H_
#define _MDB_MIDL_H_
#include <stddef.h>
#ifdef __cplusplus
extern "C" {
#endif
/** @defgroup internal LMDB Internals
* @{
*/
/** @defgroup idls ID List Management
* @{
*/
/** A generic unsigned ID number. These were entryIDs in back-bdb.
* Preferably it should have the same size as a pointer.
*/
@ -69,81 +27,16 @@ typedef MDB_ID *MDB_IDL;
#define MDB_IDL_FIRST(ids) ((ids)[1])
#define MDB_IDL_LAST(ids) ((ids)[(ids)[0]])
/** Current max length of an #mdb_midl_alloc()ed IDL */
/** Current max length of an #mdbx_midl_alloc()ed IDL */
#define MDB_IDL_ALLOCLEN(ids) ((ids)[-1])
#ifdef MDBX_MODE_ENABLED
/** Append ID to IDL. The IDL must be big enough. */
#define mdb_midl_xappend(idl, id) do { \
#define mdbx_midl_xappend(idl, id) \
do { \
MDB_ID *xidl = (idl), xlen = ++(xidl[0]); \
xidl[xlen] = (id); \
} while (0)
/** Search for an ID in an IDL.
* @param[in] ids The IDL to search.
* @param[in] id The ID to search for.
* @return The index of the first ID greater than or equal to \b id.
*/
static unsigned mdb_midl_search( MDB_IDL ids, MDB_ID id );
/** Allocate an IDL.
* Allocates memory for an IDL of the given size.
* @return IDL on success, NULL on failure.
*/
static MDB_IDL mdb_midl_alloc(int num);
/** Free an IDL.
* @param[in] ids The IDL to free.
*/
static void mdb_midl_free(MDB_IDL ids);
/** Shrink an IDL.
* Return the IDL to the default size if it has grown larger.
* @param[in,out] idp Address of the IDL to shrink.
*/
static void mdb_midl_shrink(MDB_IDL *idp);
/** Make room for num additional elements in an IDL.
* @param[in,out] idp Address of the IDL.
* @param[in] num Number of elements to make room for.
* @return 0 on success, ENOMEM on failure.
*/
static int mdb_midl_need(MDB_IDL *idp, unsigned num);
/** Append an ID onto an IDL.
* @param[in,out] idp Address of the IDL to append to.
* @param[in] id The ID to append.
* @return 0 on success, ENOMEM if the IDL is too large.
*/
static int mdb_midl_append( MDB_IDL *idp, MDB_ID id );
/** Append an IDL onto an IDL.
* @param[in,out] idp Address of the IDL to append to.
* @param[in] app The IDL to append.
* @return 0 on success, ENOMEM if the IDL is too large.
*/
static int mdb_midl_append_list( MDB_IDL *idp, MDB_IDL app );
/** Append an ID range onto an IDL.
* @param[in,out] idp Address of the IDL to append to.
* @param[in] id The lowest ID to append.
* @param[in] n Number of IDs to append.
* @return 0 on success, ENOMEM if the IDL is too large.
*/
static int mdb_midl_append_range( MDB_IDL *idp, MDB_ID id, unsigned n );
/** Merge an IDL onto an IDL. The destination IDL must be big enough.
* @param[in] idl The IDL to merge into.
* @param[in] merge The IDL to merge.
*/
static void mdb_midl_xmerge( MDB_IDL idl, MDB_IDL merge );
/** Sort an IDL.
* @param[in,out] ids The IDL to sort.
*/
static void mdb_midl_sort( MDB_IDL ids );
/** An ID2 is an ID/pointer pair.
*/
typedef struct MDB_ID2 {
@ -153,38 +46,8 @@ typedef struct MDB_ID2 {
/** An ID2L is an ID2 List, a sorted array of ID2s.
* The first element's \b mid member is a count of how many actual
* elements are in the array. The \b mptr member of the first element is unused.
* elements are in the array. The \b mptr member of the first element is
* unused.
* The array is sorted in ascending order by \b mid.
*/
typedef MDB_ID2 *MDB_ID2L;
/** Search for an ID in an ID2L.
* @param[in] ids The ID2L to search.
* @param[in] id The ID to search for.
* @return The index of the first ID2 whose \b mid member is greater than or equal to \b id.
*/
static unsigned mdb_mid2l_search( MDB_ID2L ids, MDB_ID id );
/** Insert an ID2 into a ID2L.
* @param[in,out] ids The ID2L to insert into.
* @param[in] id The ID2 to insert.
* @return 0 on success, -1 if the ID was already present in the ID2L.
*/
static int mdb_mid2l_insert( MDB_ID2L ids, MDB_ID2 *id );
/** Append an ID2 into a ID2L.
* @param[in,out] ids The ID2L to append into.
* @param[in] id The ID2 to append.
* @return 0 on success, -2 if the ID2L is too big.
*/
static int mdb_mid2l_append( MDB_ID2L ids, MDB_ID2 *id );
#endif /* #ifdef MDBX_MODE_ENABLED */
/** @} */
/** @} */
#ifdef __cplusplus
}
#endif
#endif /* _MDB_MIDL_H_ */

159
mtest0.c
View File

@ -15,45 +15,45 @@
* <http://www.OpenLDAP.org/license.html>.
*/
#include "mdbx.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>
#include <sys/stat.h>
#include "mdbx.h"
#include <pthread.h>
#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
#define CHECK(test, msg) \
((test) ? (void)0 : ((void)fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, \
__LINE__, msg, mdbx_strerror(rc)), \
abort()))
#ifndef DBPATH
#define DBPATH "./testdb"
#endif
void* thread_entry(void *ctx)
{
void *thread_entry(void *ctx) {
MDB_env *env = ctx;
MDB_txn *txn;
int rc;
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
mdb_txn_abort(txn);
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
mdbx_txn_abort(txn);
return NULL;
}
int main(int argc,char * argv[])
{
int main(int argc, char *argv[]) {
int i = 0, j = 0, rc;
MDB_env *env;
MDB_dbi dbi;
MDB_val key, data;
MDB_txn *txn;
MDB_stat mst;
MDBX_stat mst;
MDB_cursor *cursor, *cur2;
MDB_cursor_op op;
int count;
@ -73,9 +73,9 @@ int main(int argc,char * argv[])
values[i] = rand() % 1024;
}
E(mdb_env_create(&env));
E(mdb_env_set_maxreaders(env, 42));
E(mdb_env_set_mapsize(env, 10485760));
E(mdbx_env_create(&env));
E(mdbx_env_set_maxreaders(env, 42));
E(mdbx_env_set_mapsize(env, 10485760));
E(stat("/proc/self/exe", &exe_stat) ? errno : 0);
E(stat(DBPATH "/.", &db_stat) ? errno : 0);
@ -84,14 +84,15 @@ int main(int argc,char * argv[])
/* LY: Assume running inside a CI-environment:
* 1) don't use FIXEDMAP to avoid EBUSY in case collision,
* which could be inspired by address space randomisation feature.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some dedicated storage.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some
* dedicated storage.
*/
env_oflags = 0;
}
E(mdb_env_open(env, DBPATH, env_oflags, 0664));
E(mdbx_env_open(env, DBPATH, env_oflags, 0664));
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdb_dbi_open(txn, NULL, 0, &dbi));
E(mdbx_txn_begin(env, NULL, 0, &txn));
E(mdbx_dbi_open(txn, NULL, 0, &dbi));
key.mv_size = sizeof(int);
key.mv_data = sval;
@ -102,101 +103,97 @@ int main(int argc,char * argv[])
/* Set <data> in each iteration, since MDB_NOOVERWRITE may modify it */
data.mv_size = sizeof(sval);
data.mv_data = sval;
if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NOOVERWRITE))) {
if (RES(MDB_KEYEXIST, mdbx_put(txn, dbi, &key, &data, MDB_NOOVERWRITE))) {
j++;
data.mv_size = sizeof(sval);
data.mv_data = sval;
}
}
if (j) printf("%d duplicates skipped\n", j);
E(mdb_txn_commit(txn));
E(mdb_env_stat(env, &mst));
if (j)
printf("%d duplicates skipped\n", j);
E(mdbx_txn_commit(txn));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int)key.mv_size,
(char *)key.mv_data, data.mv_data, (int)data.mv_size,
(char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
j = 0;
key.mv_data = sval;
for (i = count - 1; i > -1; i -= (rand() % 5)) {
j++;
txn = NULL;
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdbx_txn_begin(env, NULL, 0, &txn));
sprintf(sval, "%03x ", values[i]);
if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, NULL))) {
if (RES(MDB_NOTFOUND, mdbx_del(txn, dbi, &key, NULL))) {
j--;
mdb_txn_abort(txn);
mdbx_txn_abort(txn);
} else {
E(mdb_txn_commit(txn));
E(mdbx_txn_commit(txn));
}
}
free(values);
printf("Deleted %d values\n", j);
E(mdb_env_stat(env, &mst));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
printf("Cursor next\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
printf("Cursor last\n");
E(mdb_cursor_get(cursor, &key, &data, MDB_LAST));
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
E(mdbx_cursor_get(cursor, &key, &data, MDB_LAST));
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
printf("Cursor prev\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
printf("Cursor last/prev\n");
E(mdb_cursor_get(cursor, &key, &data, MDB_LAST));
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
E(mdbx_cursor_get(cursor, &key, &data, MDB_LAST));
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
E(mdb_cursor_get(cursor, &key, &data, MDB_PREV));
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
E(mdbx_cursor_get(cursor, &key, &data, MDB_PREV));
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
printf("Deleting with cursor\n");
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdb_cursor_open(txn, dbi, &cur2));
E(mdbx_txn_begin(env, NULL, 0, &txn));
E(mdbx_cursor_open(txn, dbi, &cur2));
for (i = 0; i < 50; i++) {
if (RES(MDB_NOTFOUND, mdb_cursor_get(cur2, &key, &data, MDB_NEXT)))
if (RES(MDB_NOTFOUND, mdbx_cursor_get(cur2, &key, &data, MDB_NEXT)))
break;
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
E(mdb_del(txn, dbi, &key, NULL));
printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int)key.mv_size,
(char *)key.mv_data, data.mv_data, (int)data.mv_size,
(char *)data.mv_data);
E(mdbx_del(txn, dbi, &key, NULL));
}
printf("Restarting cursor in txn\n");
for (op = MDB_FIRST, i = 0; i <= 32; op = MDB_NEXT, i++) {
if (RES(MDB_NOTFOUND, mdb_cursor_get(cur2, &key, &data, op)))
if (RES(MDB_NOTFOUND, mdbx_cursor_get(cur2, &key, &data, op)))
break;
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int)key.mv_size,
(char *)key.mv_data, data.mv_data, (int)data.mv_size,
(char *)data.mv_data);
}
mdb_cursor_close(cur2);
E(mdb_txn_commit(txn));
mdbx_cursor_close(cur2);
E(mdbx_txn_commit(txn));
for (i = 0; i < 41; ++i) {
pthread_t thread;
@ -204,20 +201,20 @@ int main(int argc,char * argv[])
}
printf("Restarting cursor outside txn\n");
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_txn_begin(env, NULL, 0, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
for (op = MDB_FIRST, i = 0; i <= 32; op = MDB_NEXT, i++) {
if (RES(MDB_NOTFOUND, mdb_cursor_get(cursor, &key, &data, op)))
if (RES(MDB_NOTFOUND, mdbx_cursor_get(cursor, &key, &data, op)))
break;
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int)key.mv_size,
(char *)key.mv_data, data.mv_data, (int)data.mv_size,
(char *)data.mv_data);
}
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
mdb_dbi_close(env, dbi);
mdb_env_close(env);
mdbx_dbi_close(env, dbi);
mdbx_env_close(env);
return 0;
}

135
mtest1.c
View File

@ -14,31 +14,32 @@
/* Based on mtest2.c - memory-mapped database tester/toy */
#include "mdbx.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>
#include <sys/stat.h>
#include "mdbx.h"
#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
#define CHECK(test, msg) \
((test) ? (void)0 : ((void)fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, \
__LINE__, msg, mdbx_strerror(rc)), \
abort()))
#ifndef DBPATH
#define DBPATH "./testdb"
#endif
int main(int argc,char * argv[])
{
int main(int argc, char *argv[]) {
int i = 0, j = 0, rc;
MDB_env *env;
MDB_dbi dbi;
MDB_val key, data;
MDB_txn *txn;
MDB_stat mst;
MDBX_stat mst;
MDB_cursor *cursor;
int count;
int *values;
@ -57,10 +58,10 @@ int main(int argc,char * argv[])
values[i] = rand() % 1024;
}
E(mdb_env_create(&env));
E(mdb_env_set_maxreaders(env, 1));
E(mdb_env_set_mapsize(env, 10485760));
E(mdb_env_set_maxdbs(env, 4));
E(mdbx_env_create(&env));
E(mdbx_env_set_maxreaders(env, 1));
E(mdbx_env_set_mapsize(env, 10485760));
E(mdbx_env_set_maxdbs(env, 4));
E(stat("/proc/self/exe", &exe_stat) ? errno : 0);
E(stat(DBPATH "/.", &db_stat) ? errno : 0);
@ -69,19 +70,20 @@ int main(int argc,char * argv[])
/* LY: Assume running inside a CI-environment:
* 1) don't use FIXEDMAP to avoid EBUSY in case collision,
* which could be inspired by address space randomisation feature.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some dedicated storage.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some
* dedicated storage.
*/
env_oflags = 0;
}
/* LY: especially here we always needs MDB_NOSYNC
* for testing mdbx_env_close_ex() and "redo-to-steady" on open. */
env_oflags |= MDB_NOSYNC;
E(mdb_env_open(env, DBPATH, env_oflags, 0664));
E(mdbx_env_open(env, DBPATH, env_oflags, 0664));
E(mdb_txn_begin(env, NULL, 0, &txn));
if (mdb_dbi_open(txn, "id1", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdb_drop(txn, dbi, 1));
E(mdb_dbi_open(txn, "id1", MDB_CREATE, &dbi));
E(mdbx_txn_begin(env, NULL, 0, &txn));
if (mdbx_dbi_open(txn, "id1", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdbx_drop(txn, dbi, 1));
E(mdbx_dbi_open(txn, "id1", MDB_CREATE, &dbi));
key.mv_size = sizeof(int);
key.mv_data = sval;
@ -91,39 +93,40 @@ int main(int argc,char * argv[])
printf("Adding %d values\n", count);
for (i = 0; i < count; i++) {
sprintf(sval, "%03x %d foo bar", values[i], values[i]);
if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NOOVERWRITE)))
if (RES(MDB_KEYEXIST, mdbx_put(txn, dbi, &key, &data, MDB_NOOVERWRITE)))
j++;
}
if (j) printf("%d duplicates skipped\n", j);
E(mdb_txn_commit(txn));
E(mdb_env_stat(env, &mst));
if (j)
printf("%d duplicates skipped\n", j);
E(mdbx_txn_commit(txn));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
printf("check-preset-a\n");
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
int present_a = 0;
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int)key.mv_size,
(char *)key.mv_data, data.mv_data, (int)data.mv_size,
(char *)data.mv_data);
++present_a;
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
CHECK(present_a == count - j, "mismatch");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
mdb_env_sync(env, 1);
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
mdbx_env_sync(env, 1);
int deleted = 0;
key.mv_data = sval;
for (i = count - 1; i > -1; i -= (rand() % 5)) {
txn = NULL;
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdbx_txn_begin(env, NULL, 0, &txn));
sprintf(sval, "%03x ", values[i]);
if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, NULL))) {
mdb_txn_abort(txn);
if (RES(MDB_NOTFOUND, mdbx_del(txn, dbi, &key, NULL))) {
mdbx_txn_abort(txn);
} else {
E(mdb_txn_commit(txn));
E(mdbx_txn_commit(txn));
deleted++;
}
}
@ -131,69 +134,65 @@ int main(int argc,char * argv[])
printf("Deleted %d values\n", deleted);
printf("check-preset-b.cursor-next\n");
E(mdb_env_stat(env, &mst));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
int present_b = 0;
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
++present_b;
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
CHECK(present_b == present_a - deleted, "mismatch");
printf("check-preset-b.cursor-prev\n");
j = 1;
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
++j;
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
CHECK(present_b == j, "mismatch");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
mdb_dbi_close(env, dbi);
mdbx_dbi_close(env, dbi);
/********************* LY: kept DB dirty ****************/
mdbx_env_close_ex(env, 1);
E(mdb_env_create(&env));
E(mdb_env_set_maxdbs(env, 4));
E(mdb_env_open(env, DBPATH, env_oflags, 0664));
E(mdbx_env_create(&env));
E(mdbx_env_set_maxdbs(env, 4));
E(mdbx_env_open(env, DBPATH, env_oflags, 0664));
printf("check-preset-c.cursor-next\n");
E(mdb_env_stat(env, &mst));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_dbi_open(txn, "id1", 0, &dbi));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_dbi_open(txn, "id1", 0, &dbi));
E(mdbx_cursor_open(txn, dbi, &cursor));
int present_c = 0;
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
++present_c;
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
printf("Rolled back %d deletion(s)\n", present_c - (present_a - deleted));
CHECK(present_c > present_a - deleted, "mismatch");
printf("check-preset-d.cursor-prev\n");
j = 1;
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
++j;
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
CHECK(present_c == j, "mismatch");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
mdb_dbi_close(env, dbi);
mdbx_dbi_close(env, dbi);
mdbx_env_close_ex(env, 0);
return 0;

101
mtest2.c
View File

@ -17,31 +17,32 @@
/* Just like mtest.c, but using a subDB instead of the main DB */
#include "mdbx.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>
#include <sys/stat.h>
#include "mdbx.h"
#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
#define CHECK(test, msg) \
((test) ? (void)0 : ((void)fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, \
__LINE__, msg, mdbx_strerror(rc)), \
abort()))
#ifndef DBPATH
#define DBPATH "./testdb"
#endif
int main(int argc,char * argv[])
{
int main(int argc, char *argv[]) {
int i = 0, j = 0, rc;
MDB_env *env;
MDB_dbi dbi;
MDB_val key, data;
MDB_txn *txn;
MDB_stat mst;
MDBX_stat mst;
MDB_cursor *cursor;
int count;
int *values;
@ -60,10 +61,10 @@ int main(int argc,char * argv[])
values[i] = rand() % 1024;
}
E(mdb_env_create(&env));
E(mdb_env_set_maxreaders(env, 1));
E(mdb_env_set_mapsize(env, 10485760));
E(mdb_env_set_maxdbs(env, 4));
E(mdbx_env_create(&env));
E(mdbx_env_set_maxreaders(env, 1));
E(mdbx_env_set_mapsize(env, 10485760));
E(mdbx_env_set_maxdbs(env, 4));
E(stat("/proc/self/exe", &exe_stat) ? errno : 0);
E(stat(DBPATH "/.", &db_stat) ? errno : 0);
@ -72,16 +73,17 @@ int main(int argc,char * argv[])
/* LY: Assume running inside a CI-environment:
* 1) don't use FIXEDMAP to avoid EBUSY in case collision,
* which could be inspired by address space randomisation feature.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some dedicated storage.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some
* dedicated storage.
*/
env_oflags = 0;
}
E(mdb_env_open(env, DBPATH, env_oflags, 0664));
E(mdbx_env_open(env, DBPATH, env_oflags, 0664));
E(mdb_txn_begin(env, NULL, 0, &txn));
if (mdb_dbi_open(txn, "id2", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdb_drop(txn, dbi, 1));
E(mdb_dbi_open(txn, "id2", MDB_CREATE, &dbi));
E(mdbx_txn_begin(env, NULL, 0, &txn));
if (mdbx_dbi_open(txn, "id2", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdbx_drop(txn, dbi, 1));
E(mdbx_dbi_open(txn, "id2", MDB_CREATE, &dbi));
key.mv_size = sizeof(int);
key.mv_data = sval;
@ -91,62 +93,61 @@ int main(int argc,char * argv[])
sprintf(sval, "%03x %d foo bar", values[i], values[i]);
data.mv_size = sizeof(sval);
data.mv_data = sval;
if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NOOVERWRITE)))
if (RES(MDB_KEYEXIST, mdbx_put(txn, dbi, &key, &data, MDB_NOOVERWRITE)))
j++;
}
if (j) printf("%d duplicates skipped\n", j);
E(mdb_txn_commit(txn));
E(mdb_env_stat(env, &mst));
if (j)
printf("%d duplicates skipped\n", j);
E(mdbx_txn_commit(txn));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int)key.mv_size,
(char *)key.mv_data, data.mv_data, (int)data.mv_size,
(char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
j = 0;
key.mv_data = sval;
for (i = count - 1; i > -1; i -= (rand() % 5)) {
j++;
txn = NULL;
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdbx_txn_begin(env, NULL, 0, &txn));
sprintf(sval, "%03x ", values[i]);
if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, NULL))) {
if (RES(MDB_NOTFOUND, mdbx_del(txn, dbi, &key, NULL))) {
j--;
mdb_txn_abort(txn);
mdbx_txn_abort(txn);
} else {
E(mdb_txn_commit(txn));
E(mdbx_txn_commit(txn));
}
}
free(values);
printf("Deleted %d values\n", j);
E(mdb_env_stat(env, &mst));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
printf("Cursor next\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
printf("Cursor prev\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
mdb_dbi_close(env, dbi);
mdb_env_close(env);
mdbx_dbi_close(env, dbi);
mdbx_env_close(env);
return 0;
}

99
mtest3.c
View File

@ -16,32 +16,33 @@
*/
/* Tests for sorted duplicate DBs */
#include "mdbx.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>
#include <sys/stat.h>
#include "mdbx.h"
#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
#define CHECK(test, msg) \
((test) ? (void)0 : ((void)fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, \
__LINE__, msg, mdbx_strerror(rc)), \
abort()))
#ifndef DBPATH
#define DBPATH "./testdb"
#endif
int main(int argc,char * argv[])
{
int main(int argc, char *argv[]) {
int i = 0, j = 0, rc;
MDB_env *env;
MDB_dbi dbi;
MDB_val key, data;
MDB_txn *txn;
MDB_stat mst;
MDBX_stat mst;
MDB_cursor *cursor;
int count;
int *values;
@ -63,9 +64,9 @@ int main(int argc,char * argv[])
values[i] = rand() % 1024;
}
E(mdb_env_create(&env));
E(mdb_env_set_mapsize(env, 10485760));
E(mdb_env_set_maxdbs(env, 4));
E(mdbx_env_create(&env));
E(mdbx_env_set_mapsize(env, 10485760));
E(mdbx_env_set_maxdbs(env, 4));
E(stat("/proc/self/exe", &exe_stat) ? errno : 0);
E(stat(DBPATH "/.", &db_stat) ? errno : 0);
@ -74,16 +75,17 @@ int main(int argc,char * argv[])
/* LY: Assume running inside a CI-environment:
* 1) don't use FIXEDMAP to avoid EBUSY in case collision,
* which could be inspired by address space randomisation feature.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some dedicated storage.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some
* dedicated storage.
*/
env_oflags = 0;
}
E(mdb_env_open(env, DBPATH, env_oflags, 0664));
E(mdbx_env_open(env, DBPATH, env_oflags, 0664));
E(mdb_txn_begin(env, NULL, 0, &txn));
if (mdb_dbi_open(txn, "id3", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdb_drop(txn, dbi, 1));
E(mdb_dbi_open(txn, "id3", MDB_CREATE|MDB_DUPSORT, &dbi));
E(mdbx_txn_begin(env, NULL, 0, &txn));
if (mdbx_dbi_open(txn, "id3", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdbx_drop(txn, dbi, 1));
E(mdbx_dbi_open(txn, "id3", MDB_CREATE | MDB_DUPSORT, &dbi));
key.mv_size = sizeof(int);
key.mv_data = kval;
@ -95,67 +97,66 @@ int main(int argc,char * argv[])
if (!(i & 0x0f))
sprintf(kval, "%03x", values[i]);
sprintf(sval, "%03x %d foo bar", values[i], values[i]);
if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA)))
if (RES(MDB_KEYEXIST, mdbx_put(txn, dbi, &key, &data, MDB_NODUPDATA)))
j++;
}
if (j) printf("%d duplicates skipped\n", j);
E(mdb_txn_commit(txn));
E(mdb_env_stat(env, &mst));
if (j)
printf("%d duplicates skipped\n", j);
E(mdbx_txn_commit(txn));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int)key.mv_size,
(char *)key.mv_data, data.mv_data, (int)data.mv_size,
(char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
j = 0;
for (i = count - 1; i > -1; i -= (rand() % 5)) {
j++;
txn = NULL;
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdbx_txn_begin(env, NULL, 0, &txn));
sprintf(kval, "%03x", values[i & ~0x0f]);
sprintf(sval, "%03x %d foo bar", values[i], values[i]);
key.mv_size = sizeof(int);
key.mv_data = kval;
data.mv_size = sizeof(sval);
data.mv_data = sval;
if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
if (RES(MDB_NOTFOUND, mdbx_del(txn, dbi, &key, &data))) {
j--;
mdb_txn_abort(txn);
mdbx_txn_abort(txn);
} else {
E(mdb_txn_commit(txn));
E(mdbx_txn_commit(txn));
}
}
free(values);
printf("Deleted %d values\n", j);
E(mdb_env_stat(env, &mst));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
printf("Cursor next\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
printf("Cursor prev\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
mdb_dbi_close(env, dbi);
mdb_env_close(env);
mdbx_dbi_close(env, dbi);
mdbx_env_close(env);
return 0;
}

132
mtest4.c
View File

@ -16,32 +16,33 @@
*/
/* Tests for sorted duplicate DBs with fixed-size keys */
#include "mdbx.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>
#include <sys/stat.h>
#include "mdbx.h"
#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
#define CHECK(test, msg) \
((test) ? (void)0 : ((void)fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, \
__LINE__, msg, mdbx_strerror(rc)), \
abort()))
#ifndef DBPATH
#define DBPATH "./testdb"
#endif
int main(int argc,char * argv[])
{
int main(int argc, char *argv[]) {
int i = 0, j = 0, rc;
MDB_env *env;
MDB_dbi dbi;
MDB_val key, data;
MDB_txn *txn;
MDB_stat mst;
MDBX_stat mst;
MDB_cursor *cursor;
int count;
int *values;
@ -61,9 +62,9 @@ int main(int argc,char * argv[])
values[i] = i * 5;
}
E(mdb_env_create(&env));
E(mdb_env_set_mapsize(env, 10485760));
E(mdb_env_set_maxdbs(env, 4));
E(mdbx_env_create(&env));
E(mdbx_env_set_mapsize(env, 10485760));
E(mdbx_env_set_maxdbs(env, 4));
E(stat("/proc/self/exe", &exe_stat) ? errno : 0);
E(stat(DBPATH "/.", &db_stat) ? errno : 0);
@ -72,16 +73,17 @@ int main(int argc,char * argv[])
/* LY: Assume running inside a CI-environment:
* 1) don't use FIXEDMAP to avoid EBUSY in case collision,
* which could be inspired by address space randomisation feature.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some dedicated storage.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some
* dedicated storage.
*/
env_oflags = 0;
}
E(mdb_env_open(env, DBPATH, env_oflags, 0664));
E(mdbx_env_open(env, DBPATH, env_oflags, 0664));
E(mdb_txn_begin(env, NULL, 0, &txn));
if (mdb_dbi_open(txn, "id4", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdb_drop(txn, dbi, 1));
E(mdb_dbi_open(txn, "id4", MDB_CREATE|MDB_DUPSORT|MDB_DUPFIXED, &dbi));
E(mdbx_txn_begin(env, NULL, 0, &txn));
if (mdbx_dbi_open(txn, "id4", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdbx_drop(txn, dbi, 1));
E(mdbx_dbi_open(txn, "id4", MDB_CREATE | MDB_DUPSORT | MDB_DUPFIXED, &dbi));
key.mv_size = sizeof(int);
key.mv_data = kval;
@ -92,25 +94,26 @@ int main(int argc,char * argv[])
strcpy(kval, "001");
for (i = 0; i < count; i++) {
sprintf(sval, "%07x", values[i]);
if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA)))
if (RES(MDB_KEYEXIST, mdbx_put(txn, dbi, &key, &data, MDB_NODUPDATA)))
j++;
}
if (j) printf("%d duplicates skipped\n", j);
E(mdb_txn_commit(txn));
E(mdb_env_stat(env, &mst));
if (j)
printf("%d duplicates skipped\n", j);
E(mdbx_txn_commit(txn));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
/* there should be one full page of dups now.
*/
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int)key.mv_size,
(char *)key.mv_data, data.mv_data, (int)data.mv_size,
(char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
/* test all 3 branches of split code:
* 1: new key in lower half
@ -124,73 +127,70 @@ int main(int argc,char * argv[])
data.mv_data = sval;
sprintf(sval, "%07x", values[3] + 1);
E(mdb_txn_begin(env, NULL, 0, &txn));
(void)RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA));
mdb_txn_abort(txn);
E(mdbx_txn_begin(env, NULL, 0, &txn));
(void)RES(MDB_KEYEXIST, mdbx_put(txn, dbi, &key, &data, MDB_NODUPDATA));
mdbx_txn_abort(txn);
sprintf(sval, "%07x", values[255] + 1);
E(mdb_txn_begin(env, NULL, 0, &txn));
(void)RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA));
mdb_txn_abort(txn);
E(mdbx_txn_begin(env, NULL, 0, &txn));
(void)RES(MDB_KEYEXIST, mdbx_put(txn, dbi, &key, &data, MDB_NODUPDATA));
mdbx_txn_abort(txn);
sprintf(sval, "%07x", values[500] + 1);
E(mdb_txn_begin(env, NULL, 0, &txn));
(void)RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA));
E(mdb_txn_commit(txn));
E(mdbx_txn_begin(env, NULL, 0, &txn));
(void)RES(MDB_KEYEXIST, mdbx_put(txn, dbi, &key, &data, MDB_NODUPDATA));
E(mdbx_txn_commit(txn));
/* Try MDB_NEXT_MULTIPLE */
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_MULTIPLE)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
E(mdbx_txn_begin(env, NULL, 0, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT_MULTIPLE)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
j = 0;
for (i = count - 1; i > -1; i -= (rand() % 3)) {
j++;
txn = NULL;
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdbx_txn_begin(env, NULL, 0, &txn));
sprintf(sval, "%07x", values[i]);
key.mv_size = sizeof(int);
key.mv_data = kval;
data.mv_size = sizeof(sval);
data.mv_data = sval;
if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
if (RES(MDB_NOTFOUND, mdbx_del(txn, dbi, &key, &data))) {
j--;
mdb_txn_abort(txn);
mdbx_txn_abort(txn);
} else {
E(mdb_txn_commit(txn));
E(mdbx_txn_commit(txn));
}
}
free(values);
printf("Deleted %d values\n", j);
E(mdb_env_stat(env, &mst));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
printf("Cursor next\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
printf("Cursor prev\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
mdb_dbi_close(env, dbi);
mdb_env_close(env);
mdbx_dbi_close(env, dbi);
mdbx_env_close(env);
return 0;
}

103
mtest5.c
View File

@ -16,32 +16,33 @@
*/
/* Tests for sorted duplicate DBs using cursor_put */
#include "mdbx.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>
#include <sys/stat.h>
#include "mdbx.h"
#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
#define CHECK(test, msg) \
((test) ? (void)0 : ((void)fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, \
__LINE__, msg, mdbx_strerror(rc)), \
abort()))
#ifndef DBPATH
#define DBPATH "./testdb"
#endif
int main(int argc,char * argv[])
{
int main(int argc, char *argv[]) {
int i = 0, j = 0, rc;
MDB_env *env;
MDB_dbi dbi;
MDB_val key, data;
MDB_txn *txn;
MDB_stat mst;
MDBX_stat mst;
MDB_cursor *cursor;
int count;
int *values;
@ -63,9 +64,9 @@ int main(int argc,char * argv[])
values[i] = rand() % 1024;
}
E(mdb_env_create(&env));
E(mdb_env_set_mapsize(env, 10485760));
E(mdb_env_set_maxdbs(env, 4));
E(mdbx_env_create(&env));
E(mdbx_env_set_mapsize(env, 10485760));
E(mdbx_env_set_maxdbs(env, 4));
E(stat("/proc/self/exe", &exe_stat) ? errno : 0);
E(stat(DBPATH "/.", &db_stat) ? errno : 0);
@ -74,17 +75,18 @@ int main(int argc,char * argv[])
/* LY: Assume running inside a CI-environment:
* 1) don't use FIXEDMAP to avoid EBUSY in case collision,
* which could be inspired by address space randomisation feature.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some dedicated storage.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some
* dedicated storage.
*/
env_oflags = 0;
}
E(mdb_env_open(env, DBPATH, env_oflags, 0664));
E(mdbx_env_open(env, DBPATH, env_oflags, 0664));
E(mdb_txn_begin(env, NULL, 0, &txn));
if (mdb_dbi_open(txn, "id5", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdb_drop(txn, dbi, 1));
E(mdb_dbi_open(txn, "id5", MDB_CREATE|MDB_DUPSORT, &dbi));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_txn_begin(env, NULL, 0, &txn));
if (mdbx_dbi_open(txn, "id5", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdbx_drop(txn, dbi, 1));
E(mdbx_dbi_open(txn, "id5", MDB_CREATE | MDB_DUPSORT, &dbi));
E(mdbx_cursor_open(txn, dbi, &cursor));
key.mv_size = sizeof(int);
key.mv_data = kval;
@ -96,68 +98,67 @@ int main(int argc,char * argv[])
if (!(i & 0x0f))
sprintf(kval, "%03x", values[i]);
sprintf(sval, "%03x %d foo bar", values[i], values[i]);
if (RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NODUPDATA)))
if (RES(MDB_KEYEXIST, mdbx_cursor_put(cursor, &key, &data, MDB_NODUPDATA)))
j++;
}
if (j) printf("%d duplicates skipped\n", j);
mdb_cursor_close(cursor);
E(mdb_txn_commit(txn));
E(mdb_env_stat(env, &mst));
if (j)
printf("%d duplicates skipped\n", j);
mdbx_cursor_close(cursor);
E(mdbx_txn_commit(txn));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int)key.mv_size,
(char *)key.mv_data, data.mv_data, (int)data.mv_size,
(char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
j = 0;
for (i = count - 1; i > -1; i -= (rand() % 5)) {
j++;
txn = NULL;
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdbx_txn_begin(env, NULL, 0, &txn));
sprintf(kval, "%03x", values[i & ~0x0f]);
sprintf(sval, "%03x %d foo bar", values[i], values[i]);
key.mv_size = sizeof(int);
key.mv_data = kval;
data.mv_size = sizeof(sval);
data.mv_data = sval;
if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
if (RES(MDB_NOTFOUND, mdbx_del(txn, dbi, &key, &data))) {
j--;
mdb_txn_abort(txn);
mdbx_txn_abort(txn);
} else {
E(mdb_txn_commit(txn));
E(mdbx_txn_commit(txn));
}
}
free(values);
printf("Deleted %d values\n", j);
E(mdb_env_stat(env, &mst));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
printf("Cursor next\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
printf("Cursor prev\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n", (int)key.mv_size, (char *)key.mv_data,
(int)data.mv_size, (char *)data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
mdb_dbi_close(env, dbi);
mdb_env_close(env);
mdbx_dbi_close(env, dbi);
mdbx_env_close(env);
return 0;
}

92
mtest6.c
View File

@ -16,19 +16,21 @@
*/
/* Tests for DB splits and merges */
#include "mdbx.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>
#include <sys/stat.h>
#include "mdbx.h"
#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
#define CHECK(test, msg) \
((test) ? (void)0 : ((void)fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, \
__LINE__, msg, mdbx_strerror(rc)), \
abort()))
#ifndef DBPATH
#define DBPATH "./testdb"
@ -36,14 +38,13 @@
char dkbuf[1024];
int main(int argc,char * argv[])
{
int main(int argc, char *argv[]) {
int i = 0, rc;
MDB_env *env;
MDB_dbi dbi;
MDB_val key, data, sdata;
MDB_txn *txn;
MDB_stat mst;
MDBX_stat mst;
MDB_cursor *cursor;
long kval;
char *sval;
@ -54,9 +55,9 @@ int main(int argc,char * argv[])
(void)argv;
srand(time(NULL));
E(mdb_env_create(&env));
E(mdb_env_set_mapsize(env, 10485760));
E(mdb_env_set_maxdbs(env, 4));
E(mdbx_env_create(&env));
E(mdbx_env_set_mapsize(env, 10485760));
E(mdbx_env_set_maxdbs(env, 4));
E(stat("/proc/self/exe", &exe_stat) ? errno : 0);
E(stat(DBPATH "/.", &db_stat) ? errno : 0);
@ -65,18 +66,19 @@ int main(int argc,char * argv[])
/* LY: Assume running inside a CI-environment:
* 1) don't use FIXEDMAP to avoid EBUSY in case collision,
* which could be inspired by address space randomisation feature.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some dedicated storage.
* 2) drop MDB_NOSYNC expecting that DBPATH is at a tmpfs or some
* dedicated storage.
*/
env_oflags = 0;
}
E(mdb_env_open(env, DBPATH, env_oflags, 0664));
E(mdbx_env_open(env, DBPATH, env_oflags, 0664));
E(mdb_txn_begin(env, NULL, 0, &txn));
if (mdb_dbi_open(txn, "id6", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdb_drop(txn, dbi, 1));
E(mdb_dbi_open(txn, "id6", MDB_CREATE|MDB_INTEGERKEY, &dbi));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdb_stat(txn, dbi, &mst));
E(mdbx_txn_begin(env, NULL, 0, &txn));
if (mdbx_dbi_open(txn, "id6", MDB_CREATE, &dbi) == MDB_SUCCESS)
E(mdbx_drop(txn, dbi, 1));
E(mdbx_dbi_open(txn, "id6", MDB_CREATE | MDB_INTEGERKEY, &dbi));
E(mdbx_cursor_open(txn, dbi, &cursor));
E(mdbx_stat(txn, dbi, &mst, sizeof(mst)));
sval = calloc(1, mst.ms_psize / 4);
key.mv_size = sizeof(long);
@ -89,32 +91,34 @@ int main(int argc,char * argv[])
kval = i * 5;
sprintf(sval, "%08lx", kval);
data = sdata;
(void)RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
(void)RES(MDB_KEYEXIST,
mdbx_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
}
printf("Adding 12 more values, should yield 3 splits\n");
for (i = 0; i < 12; i++) {
kval = i * 5 + 4;
sprintf(sval, "%08lx", kval);
data = sdata;
(void)RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
(void)RES(MDB_KEYEXIST,
mdbx_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
}
printf("Adding 12 more values, should yield 3 splits\n");
for (i = 0; i < 12; i++) {
kval = i * 5 + 1;
sprintf(sval, "%08lx", kval);
data = sdata;
(void)RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
(void)RES(MDB_KEYEXIST,
mdbx_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
}
E(mdb_cursor_get(cursor, &key, &data, MDB_FIRST));
E(mdbx_cursor_get(cursor, &key, &data, MDB_FIRST));
do {
printf("key: %p %s, data: %p %.*s\n",
key.mv_data, mdb_dkey(&key, dkbuf),
printf("key: %p %s, data: %p %.*s\n", key.mv_data, mdbx_dkey(&key, dkbuf),
data.mv_data, (int)data.mv_size, (char *)data.mv_data);
} while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0);
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_commit(txn);
} while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_commit(txn);
#if 0
int j=0;
@ -124,46 +128,46 @@ int main(int argc,char * argv[])
for (i= count - 1; i > -1; i-= (rand()%5)) {
j++;
txn=NULL;
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdbx_txn_begin(env, NULL, 0, &txn));
sprintf(kval, "%03x", values[i & ~0x0f]);
sprintf(sval, "%03x %d foo bar", values[i], values[i]);
key.mv_size = sizeof(int);
key.mv_data = kval;
data.mv_size = sizeof(sval);
data.mv_data = sval;
if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
if (RES(MDB_NOTFOUND, mdbx_del(txn, dbi, &key, &data))) {
j--;
mdb_txn_abort(txn);
mdbx_txn_abort(txn);
} else {
E(mdb_txn_commit(txn));
E(mdbx_txn_commit(txn));
}
}
free(values);
printf("Deleted %d values\n", j);
E(mdb_env_stat(env, &mst));
E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdb_cursor_open(txn, dbi, &cursor));
E(mdbx_env_stat(env, &mst, sizeof(mst)));
E(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
E(mdbx_cursor_open(txn, dbi, &cursor));
printf("Cursor next\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
(int) data.mv_size, (char *) data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
printf("Cursor prev\n");
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
printf("key: %.*s, data: %.*s\n",
(int) key.mv_size, (char *) key.mv_data,
(int) data.mv_size, (char *) data.mv_data);
}
CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
CHECK(rc == MDB_NOTFOUND, "mdbx_cursor_get");
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
mdb_dbi_close(env, dbi);
mdbx_dbi_close(env, dbi);
#endif
mdb_env_close(env);
mdbx_env_close(env);
free(sval);
return 0;

28
sample-mdb.txt
View File

@ -33,10 +33,10 @@ int main(int argc,char * argv[])
/* Note: Most error checking omitted for simplicity */
rc = mdb_env_create(&env);
rc = mdb_env_open(env, "./testdb", 0, 0664);
rc = mdb_txn_begin(env, NULL, 0, &txn);
rc = mdb_dbi_open(txn, NULL, 0, &dbi);
rc = mdbx_env_create(&env);
rc = mdbx_env_open(env, "./testdb", 0, 0664);
rc = mdbx_txn_begin(env, NULL, 0, &txn);
rc = mdbx_dbi_open(txn, NULL, 0, &dbi);
key.mv_size = sizeof(int);
key.mv_data = sval;
@ -44,23 +44,23 @@ int main(int argc,char * argv[])
data.mv_data = sval;
sprintf(sval, "%03x %d foo bar", 32, 3141592);
rc = mdb_put(txn, dbi, &key, &data, 0);
rc = mdb_txn_commit(txn);
rc = mdbx_put(txn, dbi, &key, &data, 0);
rc = mdbx_txn_commit(txn);
if (rc) {
fprintf(stderr, "mdb_txn_commit: (%d) %s\n", rc, mdb_strerror(rc));
fprintf(stderr, "mdbx_txn_commit: (%d) %s\n", rc, mdbx_strerror(rc));
goto leave;
}
rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
rc = mdb_cursor_open(txn, dbi, &cursor);
while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
rc = mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn);
rc = mdbx_cursor_open(txn, dbi, &cursor);
while ((rc = mdbx_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
printf("key: %p %.*s, data: %p %.*s\n",
key.mv_data, (int) key.mv_size, (char *) key.mv_data,
data.mv_data, (int) data.mv_size, (char *) data.mv_data);
}
mdb_cursor_close(cursor);
mdb_txn_abort(txn);
mdbx_cursor_close(cursor);
mdbx_txn_abort(txn);
leave:
mdb_dbi_close(env, dbi);
mdb_env_close(env);
mdbx_dbi_close(env, dbi);
mdbx_env_close(env);
return 0;
}

101
wbench.c
View File

@ -12,22 +12,24 @@
* <http://www.OpenLDAP.org/license.html>.
*/
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/types.h>
#include <time.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <errno.h>
#include "mdbx.h"
#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
#define CHECK(test, msg) \
((test) ? (void)0 : ((void)fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, \
__LINE__, msg, mdbx_strerror(rc)), \
abort()))
#ifndef DBPATH
#define DBPATH "./testdb"
@ -38,8 +40,7 @@ struct t0 {
struct timespec ts;
};
void t0(struct t0 *t0)
{
void t0(struct t0 *t0) {
int rc;
E(getrusage(RUSAGE_SELF, &t0->ru));
E(clock_gettime(CLOCK_MONOTONIC_RAW, &t0->ts));
@ -50,20 +51,17 @@ struct info {
long iops_r, iops_w, iops_pf;
};
double delta_s(const struct timeval *begin, const struct timeval *end)
{
return end->tv_sec - begin->tv_sec
+ (end->tv_usec - begin->tv_usec) / 1000000.0;
double delta_s(const struct timeval *begin, const struct timeval *end) {
return end->tv_sec - begin->tv_sec +
(end->tv_usec - begin->tv_usec) / 1000000.0;
}
double delta2_s(const struct timespec *begin, const struct timespec *end)
{
return end->tv_sec - begin->tv_sec
+ (end->tv_nsec - begin->tv_nsec) / 1000000000.0;
double delta2_s(const struct timespec *begin, const struct timespec *end) {
return end->tv_sec - begin->tv_sec +
(end->tv_nsec - begin->tv_nsec) / 1000000000.0;
}
void measure(const struct t0 *t0, struct info *i)
{
void measure(const struct t0 *t0, struct info *i) {
struct t0 t1;
int rc;
@ -75,20 +73,18 @@ void measure(const struct t0 *t0, struct info *i)
i->cpu_sys_s = delta_s(&t0->ru.ru_stime, &t1.ru.ru_stime);
i->iops_r = t1.ru.ru_inblock - t0->ru.ru_inblock;
i->iops_w = t1.ru.ru_oublock - t0->ru.ru_oublock;
i->iops_pf = t1.ru.ru_majflt - t0->ru.ru_majflt
+ t1.ru.ru_minflt - t0->ru.ru_minflt;
i->iops_pf =
t1.ru.ru_majflt - t0->ru.ru_majflt + t1.ru.ru_minflt - t0->ru.ru_minflt;
}
void print(struct info *i)
{
void print(struct info *i) {
printf("wall-clock %.3f, iops: %lu reads, %lu writes, %lu page-faults, "
"cpu: %.3f user, %.3f sys\n",
i->wall_s, i->iops_r, i->iops_w, i->iops_pf, i->cpu_user_s, i->cpu_sys_s);
i->wall_s, i->iops_r, i->iops_w, i->iops_pf, i->cpu_user_s,
i->cpu_sys_s);
}
static void wbench(int flags, int mb, int count, int salt)
{
static void wbench(int flags, int mb, int count, int salt) {
MDB_env *env;
MDB_dbi dbi;
MDB_txn *txn;
@ -120,9 +116,9 @@ static void wbench(int flags, int mb, int count, int salt)
#endif
printf(" 0x%X\n", flags);
E(mdb_env_create(&env));
E(mdb_env_set_mapsize(env, (1ull << 20) * mb));
E(mdb_env_open(env, DBPATH, flags, 0664));
E(mdbx_env_create(&env));
E(mdbx_env_set_mapsize(env, (1ull << 20) * mb));
E(mdbx_env_open(env, DBPATH, flags, 0664));
key.mv_size = sizeof(key_value);
key.mv_data = &key_value;
@ -134,12 +130,12 @@ static void wbench(int flags, int mb, int count, int salt)
key_value = salt;
t0(&start);
for (i = 0; i < count; ++i) {
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdb_dbi_open(txn, NULL, 0, &dbi));
E(mdbx_txn_begin(env, NULL, 0, &txn));
E(mdbx_dbi_open(txn, NULL, 0, &dbi));
snprintf(data_value, sizeof(data_value), "value=%u", key_value);
E(mdb_put(txn, dbi, &key, &data, MDB_NOOVERWRITE));
E(mdb_txn_commit(txn));
E(mdbx_put(txn, dbi, &key, &data, MDB_NOOVERWRITE));
E(mdbx_txn_commit(txn));
key_value = key_value * 1664525 + 1013904223;
}
@ -151,11 +147,11 @@ static void wbench(int flags, int mb, int count, int salt)
key_value = salt;
t0(&start);
for (i = 0; i < count; ++i) {
E(mdb_txn_begin(env, NULL, 0, &txn));
E(mdb_dbi_open(txn, NULL, 0, &dbi));
E(mdbx_txn_begin(env, NULL, 0, &txn));
E(mdbx_dbi_open(txn, NULL, 0, &dbi));
E(mdb_del(txn, dbi, &key, NULL));
E(mdb_txn_commit(txn));
E(mdbx_del(txn, dbi, &key, NULL));
E(mdbx_txn_commit(txn));
key_value = key_value * 1664525 + 1013904223;
}
@ -165,11 +161,11 @@ static void wbench(int flags, int mb, int count, int salt)
printf("\tCheckpoint...");
fflush(stdout);
t0(&start);
mdb_env_sync(env, 1);
mdbx_env_sync(env, 1);
measure(&start, &rs);
print(&rs);
mdb_env_close(env);
mdbx_env_close(env);
rt.wall_s = ra.wall_s + rd.wall_s + rs.wall_s;
rt.cpu_sys_s = ra.cpu_sys_s + rd.cpu_sys_s + rs.cpu_sys_s;
rt.cpu_user_s = ra.cpu_user_s + rd.cpu_user_s + rs.cpu_user_s;
@ -194,12 +190,11 @@ static void wbench(int flags, int mb, int count, int salt)
if (flags & MDBX_LIFORECLAIM)
fprintf(stderr, " LIFO");
#endif
fprintf(stderr, "\t%.3f\t%.3f\t%.3f\t%.3f\n", rt.iops_w / 1000.0, rt.cpu_user_s, rt.cpu_sys_s, rt.wall_s);
fprintf(stderr, "\t%.3f\t%.3f\t%.3f\t%.3f\n", rt.iops_w / 1000.0,
rt.cpu_user_s, rt.cpu_sys_s, rt.wall_s);
}
int main(int argc,char * argv[])
{
int main(int argc, char *argv[]) {
(void)argc;
(void)argv;
@ -227,7 +222,8 @@ int main(int argc,char * argv[])
wbench(MDB_NOSYNC, SIZE, COUNT, SALT);
#if defined(MDBX_COALESCE) && defined(MDBX_LIFORECLAIM)
// wbench(MDB_NOSYNC | MDBX_COALESCE, SIZE, COUNT, SALT);
// wbench(MDB_NOSYNC | MDBX_COALESCE | MDBX_LIFORECLAIM, SIZE, COUNT, SALT);
// wbench(MDB_NOSYNC | MDBX_COALESCE | MDBX_LIFORECLAIM, SIZE, COUNT,
// SALT);
// wbench(MDB_NOSYNC | MDBX_LIFORECLAIM, SIZE, COUNT, SALT);
#endif
@ -242,17 +238,22 @@ int main(int argc,char * argv[])
printf("\nasync...");
wbench(MDB_WRITEMAP | MDB_MAPASYNC, SIZE, COUNT, SALT);
#if defined(MDBX_COALESCE) && defined(MDBX_LIFORECLAIM)
// wbench(MDB_WRITEMAP | MDB_MAPASYNC | MDBX_COALESCE, SIZE, COUNT, SALT);
wbench(MDB_WRITEMAP | MDB_MAPASYNC | MDBX_COALESCE | MDBX_LIFORECLAIM, SIZE, COUNT, SALT);
// wbench(MDB_WRITEMAP | MDB_MAPASYNC | MDBX_LIFORECLAIM, SIZE, COUNT, SALT);
// wbench(MDB_WRITEMAP | MDB_MAPASYNC | MDBX_COALESCE, SIZE, COUNT,
// SALT);
wbench(MDB_WRITEMAP | MDB_MAPASYNC | MDBX_COALESCE | MDBX_LIFORECLAIM, SIZE,
COUNT, SALT);
// wbench(MDB_WRITEMAP | MDB_MAPASYNC | MDBX_LIFORECLAIM, SIZE, COUNT,
// SALT);
#endif
printf("\nr/w-map + no-sync...");
wbench(MDB_NOSYNC | MDB_WRITEMAP, SIZE, COUNT, SALT);
#if defined(MDBX_COALESCE) && defined(MDBX_LIFORECLAIM)
// wbench(MDB_NOSYNC | MDB_WRITEMAP | MDBX_COALESCE, SIZE, COUNT, SALT);
wbench(MDB_NOSYNC | MDB_WRITEMAP | MDBX_COALESCE | MDBX_LIFORECLAIM, SIZE, COUNT, SALT);
// wbench(MDB_NOSYNC | MDB_WRITEMAP | MDBX_LIFORECLAIM, SIZE, COUNT, SALT);
wbench(MDB_NOSYNC | MDB_WRITEMAP | MDBX_COALESCE | MDBX_LIFORECLAIM, SIZE,
COUNT, SALT);
// wbench(MDB_NOSYNC | MDB_WRITEMAP | MDBX_LIFORECLAIM, SIZE, COUNT,
// SALT);
#endif
return 0;

145
yota_test1.c
View File

@ -1,6 +1,7 @@
/*
* Copyright 2016-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2015 Vladimir Romanov <https://www.linkedin.com/in/vladimirromanov>, Yota Lab.
* Copyright 2015 Vladimir Romanov
* <https://www.linkedin.com/in/vladimirromanov>, Yota Lab.
*
* This file is part of libmdbx.
*
@ -18,28 +19,30 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <sys/time.h>
#include <sys/stat.h>
#include <sys/time.h>
#include "mdbx.h"
#include <assert.h>
#include <limits.h>
#include <stddef.h>
#include <string.h>
#include <stdio.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include "mdbx.h"
#include <string.h>
#define IP_PRINTF_ARG_HOST(addr) (int)((addr) >> 24), (int)((addr) >> 16 & 0xff), (int)((addr) >> 8 & 0xff), (int)((addr) & 0xff)
#define IP_PRINTF_ARG_HOST(addr) \
(int)((addr) >> 24), (int)((addr) >> 16 & 0xff), (int)((addr) >> 8 & 0xff), \
(int)((addr)&0xff)
char opt_db_path[PATH_MAX] = "/dev/shm/lmdb_bench1";
char opt_db_path[PATH_MAX] = "/dev/shm/x_bench1";
static MDB_env *env;
#define REC_COUNT 1000000
int64_t ids[REC_COUNT + REC_COUNT / 10];
int32_t ids_count = 0;
int64_t lmdb_add = 0;
int64_t lmdb_del = 0;
int64_t x_add = 0;
int64_t x_del = 0;
int64_t obj_id = 0;
static void add_id_to_pool(int64_t id) {
@ -68,19 +71,23 @@ static int64_t get_id_from_pool() {
do { \
const int rc = (x); \
if (rc != MDB_SUCCESS) { \
printf("Error [%d] %s in %s at %s:%d\n", rc, mdb_strerror(rc), #x, __FILE__, __LINE__); \
printf("Error [%d] %s in %s at %s:%d\n", rc, mdbx_strerror(rc), #x, \
__FILE__, __LINE__); \
exit(EXIT_FAILURE); \
} \
} while (0)
static void db_connect() {
LMDB_CHECK(mdb_env_create(&env));
LMDB_CHECK(mdb_env_set_mapsize(env, 3L * 1024L * 1024L * 1024L));
LMDB_CHECK(mdb_env_set_maxdbs(env, 30));
LMDB_CHECK(mdbx_env_create(&env));
LMDB_CHECK(mdbx_env_set_mapsize(env, 3L * 1024L * 1024L * 1024L));
LMDB_CHECK(mdbx_env_set_maxdbs(env, 30));
#if defined(MDBX_LIFORECLAIM)
LMDB_CHECK(mdb_env_open(env, opt_db_path, MDB_CREATE | MDB_NOSYNC | MDB_WRITEMAP | MDBX_LIFORECLAIM, 0664));
LMDB_CHECK(mdbx_env_open(
env, opt_db_path,
MDB_CREATE | MDB_NOSYNC | MDB_WRITEMAP | MDBX_LIFORECLAIM, 0664));
#else
LMDB_CHECK(mdb_env_open(env, opt_db_path, MDB_CREATE | MDB_NOSYNC | MDB_WRITEMAP, 0664));
LMDB_CHECK(mdbx_env_open(env, opt_db_path,
MDB_CREATE | MDB_NOSYNC | MDB_WRITEMAP, 0664));
#endif
printf("Connection open\n");
}
@ -106,9 +113,14 @@ static void create_record(int64_t record_id) {
MDB_txn *txn;
session_data_t data;
// transaction init
snprintf(data.session_id1, sizeof (data.session_id1), "mskugw%02ld_%02ld.gx.yota.ru;3800464060;4152;%ld", record_id % 3 + 1, record_id % 9 + 1, record_id);
snprintf(data.session_id2, sizeof (data.session_id2), "gx_service;%ld;%ld;node@spb-jsm1", record_id, record_id % 1000000000 + 99999);
snprintf(data.ip, sizeof (data.ip), "%d.%d.%d.%d", IP_PRINTF_ARG_HOST(record_id & 0xFFFFFFFF));
snprintf(data.session_id1, sizeof(data.session_id1),
"mskugw%02ld_%02ld.gx.yota.ru;3800464060;4152;%ld",
record_id % 3 + 1, record_id % 9 + 1, record_id);
snprintf(data.session_id2, sizeof(data.session_id2),
"gx_service;%ld;%ld;node@spb-jsm1", record_id,
record_id % 1000000000 + 99999);
snprintf(data.ip, sizeof(data.ip), "%d.%d.%d.%d",
IP_PRINTF_ARG_HOST(record_id & 0xFFFFFFFF));
event.obj_id = record_id;
event.event_type = 1;
@ -116,23 +128,27 @@ static void create_record(int64_t record_id) {
MDB_val _session_id2_rec = {data.session_id2, strlen(data.session_id2)};
MDB_val _ip_rec = {data.ip, strlen(data.ip)};
MDB_val _obj_id_rec = {&record_id, sizeof(record_id)};
MDB_val _data_rec = {&data, offsetof(session_data_t, fill) + (rand() % sizeof (data.fill))};
MDB_val _data_rec = {&data, offsetof(session_data_t, fill) +
(rand() % sizeof(data.fill))};
MDB_val _event_rec = {&event, sizeof(event)};
LMDB_CHECK(mdb_txn_begin(env, NULL, 0, &txn));
LMDB_CHECK(mdb_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdb_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdb_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdb_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
LMDB_CHECK(mdb_put(txn, dbi_session, &_obj_id_rec, &_data_rec, MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdb_put(txn, dbi_session_id, &_session_id1_rec, &_obj_id_rec, MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdb_put(txn, dbi_session_id, &_session_id2_rec, &_obj_id_rec, MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdb_put(txn, dbi_ip, &_ip_rec, &_obj_id_rec, 0));
LMDB_CHECK(mdb_put(txn, dbi_event, &_event_rec, &_obj_id_rec, 0));
LMDB_CHECK(mdbx_txn_begin(env, NULL, 0, &txn));
LMDB_CHECK(mdbx_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdbx_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdbx_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdbx_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
LMDB_CHECK(mdbx_put(txn, dbi_session, &_obj_id_rec, &_data_rec,
MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdbx_put(txn, dbi_session_id, &_session_id1_rec, &_obj_id_rec,
MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdbx_put(txn, dbi_session_id, &_session_id2_rec, &_obj_id_rec,
MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdbx_put(txn, dbi_ip, &_ip_rec, &_obj_id_rec, 0));
LMDB_CHECK(mdbx_put(txn, dbi_event, &_event_rec, &_obj_id_rec, 0));
// transaction commit
LMDB_CHECK(mdb_txn_commit(txn));
lmdb_add++;
LMDB_CHECK(mdbx_txn_commit(txn));
x_add++;
}
static void delete_record(int64_t record_id) {
@ -144,57 +160,54 @@ static void delete_record(int64_t record_id) {
MDB_txn *txn;
// transaction init
LMDB_CHECK(mdb_txn_begin(env, NULL, 0, &txn));
LMDB_CHECK(mdbx_txn_begin(env, NULL, 0, &txn));
// open database in read-write mode
LMDB_CHECK(mdb_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdb_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdb_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdb_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
LMDB_CHECK(mdbx_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdbx_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdbx_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdbx_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
// put data
MDB_val _obj_id_rec = {&record_id, sizeof(record_id)};
MDB_val v_rec;
// get data
LMDB_CHECK(mdb_get(txn, dbi_session, &_obj_id_rec, &v_rec));
LMDB_CHECK(mdbx_get(txn, dbi_session, &_obj_id_rec, &v_rec));
session_data_t *data = (session_data_t *)v_rec.mv_data;
MDB_val _session_id1_rec = {data->session_id1, strlen(data->session_id1)};
MDB_val _session_id2_rec = {data->session_id2, strlen(data->session_id2)};
MDB_val _ip_rec = {data->ip, strlen(data->ip)};
LMDB_CHECK(mdb_del(txn, dbi_session_id, &_session_id1_rec, NULL));
LMDB_CHECK(mdb_del(txn, dbi_session_id, &_session_id2_rec, NULL));
LMDB_CHECK(mdb_del(txn, dbi_ip, &_ip_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_session_id, &_session_id1_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_session_id, &_session_id2_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_ip, &_ip_rec, NULL));
event.obj_id = record_id;
event.event_type = 1;
MDB_val _event_rec = {&event, sizeof(event)};
LMDB_CHECK(mdb_del(txn, dbi_event, &_event_rec, NULL));
LMDB_CHECK(mdb_del(txn, dbi_session, &_obj_id_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_event, &_event_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_session, &_obj_id_rec, NULL));
// transaction commit
LMDB_CHECK(mdb_txn_commit(txn));
lmdb_del++;
LMDB_CHECK(mdbx_txn_commit(txn));
x_del++;
}
static void db_disconnect() {
mdb_env_close(env);
mdbx_env_close(env);
printf("Connection closed\n");
}
static void get_db_stat(const char* db, int64_t* ms_branch_pages, int64_t* ms_leaf_pages) {
static void get_db_stat(const char *db, int64_t *ms_branch_pages,
int64_t *ms_leaf_pages) {
MDB_txn *txn;
MDB_stat stat;
MDBX_stat stat;
MDB_dbi dbi;
LMDB_CHECK(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
LMDB_CHECK(mdb_dbi_open(txn, db, MDB_CREATE, &dbi));
LMDB_CHECK(mdb_stat(txn, dbi, &stat));
mdb_txn_abort(txn);
printf("%15s | %15ld | %5u | %10ld | %10ld | %11ld |\n",
db,
stat.ms_branch_pages,
stat.ms_depth,
stat.ms_entries,
stat.ms_leaf_pages,
stat.ms_overflow_pages);
LMDB_CHECK(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
LMDB_CHECK(mdbx_dbi_open(txn, db, MDB_CREATE, &dbi));
LMDB_CHECK(mdbx_stat(txn, dbi, &stat, sizeof(stat)));
mdbx_txn_abort(txn);
printf("%15s | %15ld | %5u | %10ld | %10ld | %11ld |\n", db,
stat.ms_branch_pages, stat.ms_depth, stat.ms_entries,
stat.ms_leaf_pages, stat.ms_overflow_pages);
(*ms_branch_pages) += stat.ms_branch_pages;
(*ms_leaf_pages) += stat.ms_leaf_pages;
}
@ -202,22 +215,26 @@ static void get_db_stat(const char* db, int64_t* ms_branch_pages, int64_t* ms_le
static void periodic_stat(void) {
int64_t ms_branch_pages = 0;
int64_t ms_leaf_pages = 0;
printf(" Name | ms_branch_pages | depth | entries | leaf_pages | overf_pages |\n");
printf(" Name | ms_branch_pages | depth | entries | "
"leaf_pages | overf_pages |\n");
get_db_stat("session", &ms_branch_pages, &ms_leaf_pages);
get_db_stat("session_id", &ms_branch_pages, &ms_leaf_pages);
get_db_stat("event", &ms_branch_pages, &ms_leaf_pages);
get_db_stat("ip", &ms_branch_pages, &ms_leaf_pages);
printf("%15s | %15ld | %5s | %10s | %10ld | %11s |\n", "", ms_branch_pages, "", "", ms_leaf_pages, "");
printf("%15s | %15ld | %5s | %10s | %10ld | %11s |\n", "", ms_branch_pages,
"", "", ms_leaf_pages, "");
static int64_t prev_add;
static int64_t prev_del;
static int64_t t = -1;
if (t > 0) {
int64_t delta = getTimeMicroseconds() - t;
printf("CPS: add %ld, delete %ld, items processed - %ld\n", (lmdb_add - prev_add)*1000000 / delta, (lmdb_del - prev_del)*1000000 / delta, obj_id);
printf("CPS: add %ld, delete %ld, items processed - %ld\n",
(x_add - prev_add) * 1000000 / delta,
(x_del - prev_del) * 1000000 / delta, obj_id);
}
t = getTimeMicroseconds();
prev_add = lmdb_add;
prev_del = lmdb_del;
prev_add = x_add;
prev_del = x_del;
}
static void periodic_add_rec() {

179
yota_test2.c
View File

@ -1,6 +1,7 @@
/*
* Copyright 2016-2017 Leonid Yuriev <leo@yuriev.ru>.
* Copyright 2015 Vladimir Romanov <https://www.linkedin.com/in/vladimirromanov>, Yota Lab.
* Copyright 2015 Vladimir Romanov
* <https://www.linkedin.com/in/vladimirromanov>, Yota Lab.
*
* This file is part of libmdbx.
*
@ -18,31 +19,33 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <sys/time.h>
#include <sys/stat.h>
#include <sys/time.h>
#include "mdbx.h"
#include <assert.h>
#include <limits.h>
#include <stddef.h>
#include <string.h>
#include <stdio.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include "mdbx.h"
#include <string.h>
#define IP_PRINTF_ARG_HOST(addr) (int)((addr) >> 24), (int)((addr) >> 16 & 0xff), (int)((addr) >> 8 & 0xff), (int)((addr) & 0xff)
#define IP_PRINTF_ARG_HOST(addr) \
(int)((addr) >> 24), (int)((addr) >> 16 & 0xff), (int)((addr) >> 8 & 0xff), \
(int)((addr)&0xff)
char opt_db_path[PATH_MAX] = "/dev/shm/lmdb_bench2";
char opt_db_path[PATH_MAX] = "/dev/shm/x_bench2";
static MDB_env *env;
#define REC_COUNT 1024000
int64_t ids[REC_COUNT * 10];
int32_t ids_count = 0;
int64_t lmdb_add = 0;
int64_t lmdb_del = 0;
int64_t x_add = 0;
int64_t x_del = 0;
int64_t obj_id = 0;
int64_t lmdb_data_size = 0;
int64_t lmdb_key_size = 0;
int64_t x_data_size = 0;
int64_t x_key_size = 0;
static void add_id_to_pool(int64_t id) {
ids[ids_count] = id;
@ -70,7 +73,8 @@ static int64_t get_id_from_pool() {
do { \
const int rc = (x); \
if (rc != MDB_SUCCESS) { \
printf("Error [%d] %s in %s at %s:%d\n", rc, mdb_strerror(rc), #x, __FILE__, __LINE__); \
printf("Error [%d] %s in %s at %s:%d\n", rc, mdbx_strerror(rc), #x, \
__FILE__, __LINE__); \
exit(EXIT_FAILURE); \
} \
} while (0)
@ -81,25 +85,28 @@ static void db_connect() {
MDB_dbi dbi_event;
MDB_dbi dbi_ip;
LMDB_CHECK(mdb_env_create(&env));
LMDB_CHECK(mdb_env_set_mapsize(env, 300000L * 4096L));
LMDB_CHECK(mdb_env_set_maxdbs(env, 30));
LMDB_CHECK(mdbx_env_create(&env));
LMDB_CHECK(mdbx_env_set_mapsize(env, 300000L * 4096L));
LMDB_CHECK(mdbx_env_set_maxdbs(env, 30));
#if defined(MDBX_LIFORECLAIM)
LMDB_CHECK(mdb_env_open(env, opt_db_path, MDB_CREATE | MDB_NOSYNC | MDB_WRITEMAP | MDBX_LIFORECLAIM, 0664));
LMDB_CHECK(mdbx_env_open(
env, opt_db_path,
MDB_CREATE | MDB_NOSYNC | MDB_WRITEMAP | MDBX_LIFORECLAIM, 0664));
#else
LMDB_CHECK(mdb_env_open(env, opt_db_path, MDB_CREATE | MDB_NOSYNC | MDB_WRITEMAP, 0664));
LMDB_CHECK(mdbx_env_open(env, opt_db_path,
MDB_CREATE | MDB_NOSYNC | MDB_WRITEMAP, 0664));
#endif
MDB_txn *txn;
// transaction init
LMDB_CHECK(mdb_txn_begin(env, NULL, 0, &txn));
LMDB_CHECK(mdbx_txn_begin(env, NULL, 0, &txn));
// open database in read-write mode
LMDB_CHECK(mdb_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdb_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdb_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdb_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
LMDB_CHECK(mdbx_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdbx_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdbx_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdbx_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
// transaction commit
LMDB_CHECK(mdb_txn_commit(txn));
LMDB_CHECK(mdbx_txn_commit(txn));
printf("Connection open\n");
}
@ -124,9 +131,14 @@ static void create_record(int64_t record_id) {
MDB_txn *txn;
session_data_t data;
// transaction init
snprintf(data.session_id1, sizeof (data.session_id1), "mskugw%02ld_%02ld.gx.yota.ru;3800464060;4152;%ld", record_id % 3 + 1, record_id % 9 + 1, record_id);
snprintf(data.session_id2, sizeof (data.session_id2), "gx_service;%ld;%ld;node@spb-jsm1", record_id, record_id % 1000000000 + 99999);
snprintf(data.ip, sizeof (data.ip), "%d.%d.%d.%d", IP_PRINTF_ARG_HOST(record_id & 0xFFFFFFFF));
snprintf(data.session_id1, sizeof(data.session_id1),
"mskugw%02ld_%02ld.gx.yota.ru;3800464060;4152;%ld",
record_id % 3 + 1, record_id % 9 + 1, record_id);
snprintf(data.session_id2, sizeof(data.session_id2),
"gx_service;%ld;%ld;node@spb-jsm1", record_id,
record_id % 1000000000 + 99999);
snprintf(data.ip, sizeof(data.ip), "%d.%d.%d.%d",
IP_PRINTF_ARG_HOST(record_id & 0xFFFFFFFF));
event.obj_id = record_id;
event.event_type = 1;
@ -134,25 +146,31 @@ static void create_record(int64_t record_id) {
MDB_val _session_id2_rec = {data.session_id2, strlen(data.session_id2)};
MDB_val _ip_rec = {data.ip, strlen(data.ip)};
MDB_val _obj_id_rec = {&record_id, sizeof(record_id)};
MDB_val _data_rec = {&data, offsetof(session_data_t, fill) + (rand() % sizeof (data.fill))};
MDB_val _data_rec = {&data, offsetof(session_data_t, fill) +
(rand() % sizeof(data.fill))};
MDB_val _event_rec = {&event, sizeof(event)};
LMDB_CHECK(mdb_txn_begin(env, NULL, 0, &txn));
LMDB_CHECK(mdb_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdb_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdb_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdb_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
LMDB_CHECK(mdb_put(txn, dbi_session, &_obj_id_rec, &_data_rec, MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdb_put(txn, dbi_session_id, &_session_id1_rec, &_obj_id_rec, MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdb_put(txn, dbi_session_id, &_session_id2_rec, &_obj_id_rec, MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdb_put(txn, dbi_ip, &_ip_rec, &_obj_id_rec, 0));
LMDB_CHECK(mdb_put(txn, dbi_event, &_event_rec, &_obj_id_rec, 0));
lmdb_data_size += (_data_rec.mv_size + _obj_id_rec.mv_size * 4);
lmdb_key_size += (_obj_id_rec.mv_size + _session_id1_rec.mv_size + _session_id2_rec.mv_size + _ip_rec.mv_size + _event_rec.mv_size);
LMDB_CHECK(mdbx_txn_begin(env, NULL, 0, &txn));
LMDB_CHECK(mdbx_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdbx_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdbx_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdbx_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
LMDB_CHECK(mdbx_put(txn, dbi_session, &_obj_id_rec, &_data_rec,
MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdbx_put(txn, dbi_session_id, &_session_id1_rec, &_obj_id_rec,
MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdbx_put(txn, dbi_session_id, &_session_id2_rec, &_obj_id_rec,
MDB_NOOVERWRITE | MDB_NODUPDATA));
LMDB_CHECK(mdbx_put(txn, dbi_ip, &_ip_rec, &_obj_id_rec, 0));
LMDB_CHECK(mdbx_put(txn, dbi_event, &_event_rec, &_obj_id_rec, 0));
x_data_size += (_data_rec.mv_size + _obj_id_rec.mv_size * 4);
x_key_size +=
(_obj_id_rec.mv_size + _session_id1_rec.mv_size +
_session_id2_rec.mv_size + _ip_rec.mv_size + _event_rec.mv_size);
// transaction commit
LMDB_CHECK(mdb_txn_commit(txn));
lmdb_add++;
LMDB_CHECK(mdbx_txn_commit(txn));
x_add++;
}
static void delete_record(int64_t record_id) {
@ -164,60 +182,59 @@ static void delete_record(int64_t record_id) {
MDB_txn *txn;
// transaction init
LMDB_CHECK(mdb_txn_begin(env, NULL, 0, &txn));
LMDB_CHECK(mdbx_txn_begin(env, NULL, 0, &txn));
// open database in read-write mode
LMDB_CHECK(mdb_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdb_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdb_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdb_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
LMDB_CHECK(mdbx_dbi_open(txn, "session", MDB_CREATE, &dbi_session));
LMDB_CHECK(mdbx_dbi_open(txn, "session_id", MDB_CREATE, &dbi_session_id));
LMDB_CHECK(mdbx_dbi_open(txn, "event", MDB_CREATE, &dbi_event));
LMDB_CHECK(mdbx_dbi_open(txn, "ip", MDB_CREATE, &dbi_ip));
// put data
MDB_val _obj_id_rec = {&record_id, sizeof(record_id)};
MDB_val _data_rec;
// get data
LMDB_CHECK(mdb_get(txn, dbi_session, &_obj_id_rec, &_data_rec));
LMDB_CHECK(mdbx_get(txn, dbi_session, &_obj_id_rec, &_data_rec));
session_data_t *data = (session_data_t *)_data_rec.mv_data;
MDB_val _session_id1_rec = {data->session_id1, strlen(data->session_id1)};
MDB_val _session_id2_rec = {data->session_id2, strlen(data->session_id2)};
MDB_val _ip_rec = {data->ip, strlen(data->ip)};
LMDB_CHECK(mdb_del(txn, dbi_session_id, &_session_id1_rec, NULL));
LMDB_CHECK(mdb_del(txn, dbi_session_id, &_session_id2_rec, NULL));
LMDB_CHECK(mdb_del(txn, dbi_ip, &_ip_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_session_id, &_session_id1_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_session_id, &_session_id2_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_ip, &_ip_rec, NULL));
event.obj_id = record_id;
event.event_type = 1;
MDB_val _event_rec = {&event, sizeof(event)};
LMDB_CHECK(mdb_del(txn, dbi_event, &_event_rec, NULL));
LMDB_CHECK(mdb_del(txn, dbi_session, &_obj_id_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_event, &_event_rec, NULL));
LMDB_CHECK(mdbx_del(txn, dbi_session, &_obj_id_rec, NULL));
lmdb_data_size -= (_data_rec.mv_size + _obj_id_rec.mv_size * 4);
lmdb_key_size -= (_obj_id_rec.mv_size + _session_id1_rec.mv_size + _session_id2_rec.mv_size + _ip_rec.mv_size + _event_rec.mv_size);
x_data_size -= (_data_rec.mv_size + _obj_id_rec.mv_size * 4);
x_key_size -=
(_obj_id_rec.mv_size + _session_id1_rec.mv_size +
_session_id2_rec.mv_size + _ip_rec.mv_size + _event_rec.mv_size);
// transaction commit
LMDB_CHECK(mdb_txn_commit(txn));
lmdb_del++;
LMDB_CHECK(mdbx_txn_commit(txn));
x_del++;
}
static void db_disconnect() {
mdb_env_close(env);
mdbx_env_close(env);
printf("Connection closed\n");
}
static void get_db_stat(const char* db, int64_t* ms_branch_pages, int64_t* ms_leaf_pages) {
static void get_db_stat(const char *db, int64_t *ms_branch_pages,
int64_t *ms_leaf_pages) {
MDB_txn *txn;
MDB_stat stat;
MDBX_stat stat;
MDB_dbi dbi;
LMDB_CHECK(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn));
LMDB_CHECK(mdb_dbi_open(txn, db, MDB_CREATE, &dbi));
LMDB_CHECK(mdb_stat(txn, dbi, &stat));
mdb_txn_abort(txn);
printf("%15s | %15ld | %5u | %10ld | %10ld | %11ld |\n",
db,
stat.ms_branch_pages,
stat.ms_depth,
stat.ms_entries,
stat.ms_leaf_pages,
stat.ms_overflow_pages);
LMDB_CHECK(mdbx_txn_begin(env, NULL, MDB_RDONLY, &txn));
LMDB_CHECK(mdbx_dbi_open(txn, db, MDB_CREATE, &dbi));
LMDB_CHECK(mdbx_stat(txn, dbi, &stat, sizeof(stat)));
mdbx_txn_abort(txn);
printf("%15s | %15ld | %5u | %10ld | %10ld | %11ld |\n", db,
stat.ms_branch_pages, stat.ms_depth, stat.ms_entries,
stat.ms_leaf_pages, stat.ms_overflow_pages);
(*ms_branch_pages) += stat.ms_branch_pages;
(*ms_leaf_pages) += stat.ms_leaf_pages;
}
@ -225,23 +242,31 @@ static void get_db_stat(const char* db, int64_t* ms_branch_pages, int64_t* ms_le
static void periodic_stat(void) {
int64_t ms_branch_pages = 0;
int64_t ms_leaf_pages = 0;
printf(" Name | ms_branch_pages | depth | entries | leaf_pages | overf_pages |\n");
printf(" Name | ms_branch_pages | depth | entries | "
"leaf_pages | overf_pages |\n");
get_db_stat("session", &ms_branch_pages, &ms_leaf_pages);
get_db_stat("session_id", &ms_branch_pages, &ms_leaf_pages);
get_db_stat("event", &ms_branch_pages, &ms_leaf_pages);
get_db_stat("ip", &ms_branch_pages, &ms_leaf_pages);
printf("%15s | %15ld | %5s | %10s | %10ld | %11s |\n", "", ms_branch_pages, "", "", ms_leaf_pages, "");
printf("%15s | %15ld | %5s | %10s | %10ld | %11s |\n", "", ms_branch_pages,
"", "", ms_leaf_pages, "");
static int64_t prev_add;
static int64_t prev_del;
static int64_t t = -1;
if (t > 0) {
int64_t delta = getTimeMicroseconds() - t;
printf("CPS: add %ld, delete %ld, items processed - %ldK data=%ldK key=%ldK\n", (lmdb_add - prev_add)*1000000 / delta, (lmdb_del - prev_del)*1000000 / delta, obj_id / 1024, lmdb_data_size / 1024, lmdb_key_size / 1024);
printf("usage data=%ld%%\n", ((lmdb_data_size + lmdb_key_size) * 100) / ((ms_leaf_pages + ms_branch_pages)*4096));
printf("CPS: add %ld, delete %ld, items processed - %ldK data=%ldK "
"key=%ldK\n",
(x_add - prev_add) * 1000000 / delta,
(x_del - prev_del) * 1000000 / delta, obj_id / 1024,
x_data_size / 1024, x_key_size / 1024);
printf("usage data=%ld%%\n",
((x_data_size + x_key_size) * 100) /
((ms_leaf_pages + ms_branch_pages) * 4096));
}
t = getTimeMicroseconds();
prev_add = lmdb_add;
prev_del = lmdb_del;
prev_add = x_add;
prev_del = x_del;
}
// static void periodic_add_rec() {