ISAR/libmdbx
3
0
Fork 0
mirror of https://github.com/isar/libmdbx.git synced 2024-10-29 23:19:20 +08:00
Code Issues Packages Projects Releases Wiki Activity

mdbx-doc: fix/refine README & Doxygen docs.

Browse Source 
Change-Id: I79cfb44f84fbf0f118b0d209af1ef62bb9dae72a
This commit is contained in:
Leonid Yuriev 2020-09-29 20:14:08 +03:00
parent c8a0951566
commit 72e136b9da
8 changed files with 41 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
README.md
View File

@ -12,7 +12,7 @@ libmdbx
_libmdbx_ is an extremely fast, compact, powerful, embedded,
transactional [key-value database](https://en.wikipedia.org/wiki/Key-value_database),
with [permissive license](./LICENSE).
_MDBX_ has a specific set of properties and capabilities,
_libmdbx_ has a specific set of properties and capabilities,
focused on creating unique lightweight solutions.
1. Allows **a swarm of multi-threaded processes to
@ -30,9 +30,9 @@ tree](https://en.wikipedia.org/wiki/B%2B_tree).
[WAL](https://en.wikipedia.org/wiki/Write-ahead_logging), but that might
be a caveat for write-intensive workloads with durability requirements.
4. **Compact and friendly for fully embedding**. Only 25KLOC of `C11`,
64K x86 binary code, no internal threads neither processes, but
implements a simplified variant of the [Berkeley
4. **Compact and friendly for fully embedding**. Only ≈25KLOC of `C11`,
≈64K x86 binary code of core, no internal threads neither server process(es),
but implements a simplified variant of the [Berkeley
DB](https://en.wikipedia.org/wiki/Berkeley_DB) and
[dbm](https://en.wikipedia.org/wiki/DBM_(computing)) API.
@ -51,9 +51,9 @@ OpenSolaris, OpenIndiana, NetBSD, OpenBSD and other systems compliant with
**POSIX.1-2008**.
<!-- section-end -->
Historically, _MDBX_ is a deeply revised and extended descendant of the amazing
Historically, _libmdbx_ is a deeply revised and extended descendant of the amazing
[Lightning Memory-Mapped Database](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database).
_MDBX_ inherits all benefits from _LMDB_, but resolves some issues and adds [a set of improvements](#improvements-beyond-lmdb).
_libmdbx_ inherits all benefits from _LMDB_, but resolves some issues and adds [a set of improvements](#improvements-beyond-lmdb).
<!-- section-begin mithril -->
The next version is under active non-public development from scratch and will be
@ -161,19 +161,19 @@ transaction journal. No crash recovery needed. No maintenance is required.
1. There cannot be more than one writer at a time, i.e. no more than one write transaction at a time.
2. MDBX is based on [B+ tree](https://en.wikipedia.org/wiki/B%2B_tree), so access to database pages is mostly random.
2. _libmdbx_ is based on [B+ tree](https://en.wikipedia.org/wiki/B%2B_tree), so access to database pages is mostly random.
Thus SSDs provide a significant performance boost over spinning disks for large databases.
3. MDBX uses [shadow paging](https://en.wikipedia.org/wiki/Shadow_paging) instead of [WAL](https://en.wikipedia.org/wiki/Write-ahead_logging). Thus syncing data to disk might be a bottleneck for write intensive workload.
3. _libmdbx_ uses [shadow paging](https://en.wikipedia.org/wiki/Shadow_paging) instead of [WAL](https://en.wikipedia.org/wiki/Write-ahead_logging). Thus syncing data to disk might be a bottleneck for write intensive workload.
4. MDBX uses [copy-on-write](https://en.wikipedia.org/wiki/Copy-on-write) for [snapshot isolation](https://en.wikipedia.org/wiki/Snapshot_isolation) during updates, but read transactions prevents recycling an old retired/freed pages, since it read ones. Thus altering of data during a parallel
4. _libmdbx_ uses [copy-on-write](https://en.wikipedia.org/wiki/Copy-on-write) for [snapshot isolation](https://en.wikipedia.org/wiki/Snapshot_isolation) during updates, but read transactions prevents recycling an old retired/freed pages, since it read ones. Thus altering of data during a parallel
long-lived read operation will increase the process work set, may exhaust entire free database space,
the database can grow quickly, and result in performance degradation.
Try to avoid long running read transactions.
5. MDBX is extraordinarily fast and provides minimal overhead for data access,
5. _libmdbx_ is extraordinarily fast and provides minimal overhead for data access,
so you should reconsider using brute force techniques and double check your code.
On the one hand, in the case of MDBX, a simple linear search may be more profitable than complex indexes.
On the one hand, in the case of _libmdbx_, a simple linear search may be more profitable than complex indexes.
On the other hand, if you make something suboptimally, you can notice detrimentally only on sufficiently large data.
## Comparison with other databases
@ -305,7 +305,7 @@ named mutexes are used.
# History
Historically, _MDBX_ is a deeply revised and extended descendant of the
Historically, _libmdbx_ is a deeply revised and extended descendant of the
[Lightning Memory-Mapped Database](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database).
At first the development was carried out within the
[ReOpenLDAP](https://github.com/erthink/ReOpenLDAP) project. About a
@ -318,7 +318,7 @@ and development is funded by [Positive Technologies](https://www.ptsecurity.com)
## Acknowledgments
Howard Chu <hyc@openldap.org> is the author of LMDB, from which
originated the MDBX in 2015.
originated the _libmdbx_ in 2015.
Martin Hedenfalk <martin@bzero.se> is the author of `btree.c` code, which
was used to begin development of LMDB.

2
docs/Doxyfile.in
View File

@ -44,7 +44,7 @@ PROJECT_NUMBER = "${MDBX_VERSION_MAJOR}.${MDBX_VERSION_MINOR}.${MDBX_VER
# for a project that appears at the top of each page and should give viewer a
# quick idea about the purpose of the project. Keep the description short.
PROJECT_BRIEF = "One of the fastest embeddable key-value ACID database without WAL."
PROJECT_BRIEF = "One of the fastest compact embeddable key-value ACID database without WAL."
# With the PROJECT_LOGO tag one can specify a logo or an icon that is included
# in the documentation. The maximum height of the logo should not exceed 55

2
docs/_preface.md
View File

@ -34,7 +34,7 @@ periodic checkpointing and/or compaction of their log or database files
otherwise they grow without bound. MDBX tracks retired/freed pages within the
database and re-uses them for new write operations, so the database size does
not grow without bound in normal use. It is worth noting that the "next"
version libmdbx (MithrilDB) will solve this problem.
version libmdbx (\ref MithrilDB) will solve this problem.
The memory map can be used as a read-only or read-write map. It is read-only
by default as this provides total immunity to corruption. Using read-write

34
docs/_restrictions.md
View File

@ -15,10 +15,10 @@ In addition to those listed for some functions.
multiple processes in a lock-free manner and any locking is
unwise due to a large overhead.
The "next" version of libmdbx (MithrilDB) will solve this issue.
The "next" version of libmdbx (\ref MithrilDB) will solve this issue.
\note Workaround: Just make all programs using the database close it;
the LCK-file is always reset on first open.
the LCK-file is always reset on first open.
2. Stale reader transactions left behind by an aborted program cause
further writes to grow the database quickly, and stale locks can
@ -27,7 +27,7 @@ In addition to those listed for some functions.
growth the database. But in some cases, this may not be enough.
\note Workaround: Check for stale readers periodically, using the
`mdbx_reader_check()` function or the mdbx_stat tool.
\ref mdbx_reader_check() function or the mdbx_stat tool.
3. Stale writers will be cleared automatically by MDBX on supported
platforms. But this is platform-specific, especially of
@ -36,8 +36,7 @@ In addition to those listed for some functions.
Windows and FreeBSD.
\note Workaround: Otherwise just make all programs using the database
close it; the LCK-file is always reset on first open
of the environment.
close it; the LCK-file is always reset on first open of the environment.
## Remote filesystems
@ -51,19 +50,19 @@ a read-only network shares.
## Child processes
Do not use opened `MDBX_env` instance(s) in a child processes after `fork()`.
Do not use opened \ref MDBX_env instance(s) in a child processes after `fork()`.
It would be insane to call fork() and any MDBX-functions simultaneously
from multiple threads. The best way is to prevent the presence of open
MDBX-instances during `fork()`.
The `MDBX_ENV_CHECKPID` build-time option, which is ON by default on
The \ref MDBX_ENV_CHECKPID build-time option, which is ON by default on
non-Windows platforms (i.e. where `fork()` is available), enables PID
checking at a few critical points. But this does not give any guarantees,
but only allows you to detect such errors a little sooner. Depending on
the platform, you should expect an application crash and/or database
corruption in such cases.
On the other hand, MDBX allow calling `mdbx_close_env()` in such cases to
On the other hand, MDBX allow calling \ref mdbx_env_close() in such cases to
release resources, but no more and in general this is a wrong way.
## Read-only mode
@ -72,24 +71,24 @@ readers need write access to LCK-file to be ones visible for writer.
So MDBX always tries to open/create LCK-file for read-write, but switches
to without-LCK mode on appropriate errors (`EROFS`, `EACCESS`, `EPERM`)
if the read-only mode was requested by the `MDBX_RDONLY` flag which is
if the read-only mode was requested by the \ref MDBX_RDONLY flag which is
described below.
The "next" version of libmdbx (MithrilDB) will solve this issue for the "many
The "next" version of libmdbx (\ref MithrilDB) will solve this issue for the "many
readers without writer" case.
## One thread - One transaction
A thread can only use one transaction at a time, plus any nested
read-write transactions in the non-writemap mode. Each transaction
belongs to one thread. The `MDBX_NOTLS` flag changes this for read-only
belongs to one thread. The \ref MDBX_NOTLS flag changes this for read-only
transactions. See below.
Do not start more than one transaction for a one thread. If you think
about this, it's really strange to do something with two data snapshots
at once, which may be different. MDBX checks and preventing this by
returning corresponding error code (`MDBX_TXN_OVERLAPPING`, `MDBX_BAD_RSLOT`,
`MDBX_BUSY`) unless you using `MDBX_NOTLS` option on the environment.
returning corresponding error code (\ref MDBX_TXN_OVERLAPPING, \ref MDBX_BAD_RSLOT,
\ref MDBX_BUSY) unless you using \ref MDBX_NOTLS option on the environment.
Nonetheless, with the `MDBX_NOTLS` option, you must know exactly what you
are doing, otherwise you will get deadlocks or reading an alien data.
@ -97,7 +96,7 @@ readers without writer" case.
## Do not open twice
Do not have open an MDBX database twice in the same process at the same
time. By default MDBX prevent this in most cases by tracking databases
opening and return `MDBX_BUSY` if anyone LCK-file is already open.
opening and return \ref MDBX_BUSY if anyone LCK-file is already open.
The reason for this is that when the "Open file description" locks (aka
OFD-locks) are not available, MDBX uses POSIX locks on files, and these
@ -130,7 +129,8 @@ reasonable to simplify this as follows:
discernible because of high transaction rate and intentional
internals simplification in favor of performance.
2. MDBX employs Multiversion concurrency control on the Copy-on-Write
2. MDBX employs [Multiversion concurrency control](https://en.wikipedia.org/wiki/Multiversion_concurrency_control)
on the [Copy-on-Write](https://en.wikipedia.org/wiki/Copy-on-write)
basis, that allows multiple readers runs in parallel with a write
transaction without blocking. An each write transaction needs free
pages to put the changed data, that pages will be placed in the new
@ -150,13 +150,11 @@ performance degradation.
MDBX mostly solve "long-lived" readers issue by using the Handle-Slow-Readers
\ref MDBX_hsr_func callback which allows to abort long-lived read transactions,
and using the \ref MDBX_LIFORECLAIM mode which addresses subsequent performance degradation.
The "next" version of libmdbx (MithrilDB) will completely solve this.
The "next" version of libmdbx (\ref MithrilDB) will completely solve this.
- Avoid suspending a process with active transactions. These would then be
"long-lived" as above.
The "next" version of libmdbx (MithrilDB) will solve this issue.
- Avoid aborting a process with an active read-only transaction in scenarios
with high rate of write transactions. The transaction becomes "long-lived"
as above until a check for stale readers is performed or the LCK-file is

2
docs/_toc.md
View File

@ -27,4 +27,4 @@ including creating [PR](https://help.github.com/en/github/collaborating-with-iss
---
\section mithril Mithril DB
\section MithrilDB MithrilDB

11
mdbx.h
View File

@ -3259,7 +3259,7 @@ LIBMDBX_API int mdbx_dbi_flags(MDBX_txn *txn, MDBX_dbi dbi, unsigned *flags);
* \note Use with care.
* This call is synchronized via mutex with \ref mdbx_dbi_close(), but NOT with
* other transactions running by other threads. The "next" version of libmdbx
* (MithrilDB) will solve this issue.
* (\ref MithrilDB) will solve this issue.
*
* Handles should only be closed if no other threads are going to reference
* the database handle or one of its cursors any further. Do not close a handle
@ -3562,11 +3562,6 @@ LIBMDBX_API int mdbx_del(MDBX_txn *txn, MDBX_dbi dbi, const MDBX_val *key,
* which helps to avoid errors such as: use-after-free, double-free, i.e.
* memory corruption and segfaults.
*
* \param [in] txn A transaction handle returned by \ref mdbx_txn_begin().
* \param [in] dbi A database handle returned by \ref mdbx_dbi_open().
* \param [out] cursor Address where the new \ref MDBX_cursor handle will be
* stored.
*
* \returns Created cursor handle or NULL in case out of memory. */
LIBMDBX_API MDBX_cursor *mdbx_cursor_create(void);
@ -3589,6 +3584,7 @@ LIBMDBX_API MDBX_cursor *mdbx_cursor_create(void);
*
* \param [in] txn A transaction handle returned by \ref mdbx_txn_begin().
* \param [in] dbi A database handle returned by \ref mdbx_dbi_open().
* \param [out] cursor A cursor handle returned by \ref mdbx_cursor_create().
*
* \returns A non-zero error value on failure and 0 on success,
* some possible errors are:
@ -3645,7 +3641,8 @@ LIBMDBX_API int mdbx_cursor_open(MDBX_txn *txn, MDBX_dbi dbi,
* which helps to avoid errors such as: use-after-free, double-free, i.e.
* memory corruption and segfaults.
*
* \param [in] cursor A cursor handle returned by mdbx_cursor_open(). */
* \param [in] cursor A cursor handle returned by \ref mdbx_cursor_open()
* or \ref mdbx_cursor_create(). */
LIBMDBX_API void mdbx_cursor_close(MDBX_cursor *cursor);
/** \brief Renew a cursor handle.

5
mdbx.h++
View File

@ -2051,7 +2051,7 @@ public:
/// \note Use with care.
/// This call is synchronized via mutex with other calls \ref close_map(), but
/// NOT with other transactions running by other threads. The "next" version
/// of libmdbx (MithrilDB) will solve this issue.
/// of libmdbx (\ref MithrilDB) will solve this issue.
///
/// Handles should only be closed if no other threads are going to reference
/// the database handle or one of its cursors any further. Do not close a
@ -2127,10 +2127,13 @@ public:
/// \brief Starts read (read-only) transaction.
inline txn_managed start_read() const;
/// \brief Creates but not start read transaction.
inline txn_managed prepare_read() const;
/// \brief Starts write (read-write) transaction.
inline txn_managed start_write(bool dont_wait = false);
/// \brief Tries to start write (read-write) transaction without blocking.
inline txn_managed try_start_write();
};

2
src/options.h
View File

@ -61,7 +61,7 @@
#ifndef MDBX_ENV_CHECKPID
#if defined(MADV_DONTFORK) || defined(_WIN32) || defined(_WIN64)
/* PID check could be omitted:
* - on Linux when madvise(MADV_DONTFORK) is available. i.e. after the fork()
* - on Linux when madvise(MADV_DONTFORK) is available, i.e. after the fork()
* mapped pages will not be available for child process.
* - in Windows where fork() not available. */
#define MDBX_ENV_CHECKPID 0