mirror of
https://github.com/isar/libmdbx.git
synced 2024-12-30 02:34:13 +08:00
mdbx-doc: reorder restrictions section.
This commit is contained in:
Леонид Юрьев (Leonid Yuriev)
parent
9aa53d1616
commit
3f840ecd89
@ -2,118 +2,6 @@ Restrictions & Caveats {#restrictions}
|
||||
======================
|
||||
In addition to those listed for some functions.
|
||||
|
||||
## Troubleshooting the LCK-file
|
||||
1. A broken LCK-file can cause sync issues, including appearance of
|
||||
wrong/inconsistent data for readers. When database opened in the
|
||||
cooperative read-write mode the LCK-file requires to be mapped to
|
||||
memory in read-write access. In this case it is always possible for
|
||||
stray/malfunctioned application could writes thru pointers to
|
||||
silently corrupt the LCK-file.
|
||||
|
||||
Unfortunately, there is no any portable way to prevent such
|
||||
corruption, since the LCK-file is updated concurrently by
|
||||
multiple processes in a lock-free manner and any locking is
|
||||
unwise due to a large overhead.
|
||||
|
||||
The "next" version of libmdbx (\ref MithrilDB) will solve this issue.
|
||||
|
||||
\note Workaround: Just make all programs using the database close it;
|
||||
the LCK-file is always reset on first open.
|
||||
|
||||
2. Stale reader transactions left behind by an aborted program cause
|
||||
further writes to grow the database quickly, and stale locks can
|
||||
block further operation.
|
||||
MDBX checks for stale readers while opening environment and before
|
||||
growth the database. But in some cases, this may not be enough.
|
||||
|
||||
\note Workaround: Check for stale readers periodically, using the
|
||||
\ref mdbx_reader_check() function or the mdbx_stat tool.
|
||||
|
||||
3. Stale writers will be cleared automatically by MDBX on supported
|
||||
platforms. But this is platform-specific, especially of
|
||||
implementation of shared POSIX-mutexes and support for robust
|
||||
mutexes. For instance there are no known issues on Linux, OSX,
|
||||
Windows and FreeBSD.
|
||||
|
||||
\note Workaround: Otherwise just make all programs using the database
|
||||
close it; the LCK-file is always reset on first open of the environment.
|
||||
|
||||
|
||||
## Remote filesystems
|
||||
Do not use MDBX databases on remote filesystems, even between processes
|
||||
on the same host. This breaks file locks on some platforms, possibly
|
||||
memory map sync, and certainly sync between programs on different hosts.
|
||||
|
||||
On the other hand, MDBX support the exclusive database operation over
|
||||
a network, and cooperative read-only access to the database placed on
|
||||
a read-only network shares.
|
||||
|
||||
|
||||
## Child processes
|
||||
Do not use opened \ref MDBX_env instance(s) in a child processes after `fork()`.
|
||||
It would be insane to call fork() and any MDBX-functions simultaneously
|
||||
from multiple threads. The best way is to prevent the presence of open
|
||||
MDBX-instances during `fork()`.
|
||||
|
||||
The \ref MDBX_ENV_CHECKPID build-time option, which is ON by default on
|
||||
non-Windows platforms (i.e. where `fork()` is available), enables PID
|
||||
checking at a few critical points. But this does not give any guarantees,
|
||||
but only allows you to detect such errors a little sooner. Depending on
|
||||
the platform, you should expect an application crash and/or database
|
||||
corruption in such cases.
|
||||
|
||||
On the other hand, MDBX allow calling \ref mdbx_env_close() in such cases to
|
||||
release resources, but no more and in general this is a wrong way.
|
||||
|
||||
## Read-only mode
|
||||
There is no pure read-only mode in a normal explicitly way, since
|
||||
readers need write access to LCK-file to be ones visible for writer.
|
||||
|
||||
So MDBX always tries to open/create LCK-file for read-write, but switches
|
||||
to without-LCK mode on appropriate errors (`EROFS`, `EACCESS`, `EPERM`)
|
||||
if the read-only mode was requested by the \ref MDBX_RDONLY flag which is
|
||||
described below.
|
||||
|
||||
The "next" version of libmdbx (\ref MithrilDB) will solve this issue for the "many
|
||||
readers without writer" case.
|
||||
|
||||
|
||||
## One thread - One transaction
|
||||
A thread can only use one transaction at a time, plus any nested
|
||||
read-write transactions in the non-writemap mode. Each transaction
|
||||
belongs to one thread. The \ref MDBX_NOTLS flag changes this for read-only
|
||||
transactions. See below.
|
||||
|
||||
Do not start more than one transaction for a one thread. If you think
|
||||
about this, it's really strange to do something with two data snapshots
|
||||
at once, which may be different. MDBX checks and preventing this by
|
||||
returning corresponding error code (\ref MDBX_TXN_OVERLAPPING, \ref MDBX_BAD_RSLOT,
|
||||
\ref MDBX_BUSY) unless you using \ref MDBX_NOTLS option on the environment.
|
||||
Nonetheless, with the `MDBX_NOTLS` option, you must know exactly what you
|
||||
are doing, otherwise you will get deadlocks or reading an alien data.
|
||||
|
||||
|
||||
## Do not open twice
|
||||
Do not have open an MDBX database twice in the same process at the same
|
||||
time. By default MDBX prevent this in most cases by tracking databases
|
||||
opening and return \ref MDBX_BUSY if anyone LCK-file is already open.
|
||||
|
||||
The reason for this is that when the "Open file description" locks (aka
|
||||
OFD-locks) are not available, MDBX uses POSIX locks on files, and these
|
||||
locks have issues if one process opens a file multiple times. If a single
|
||||
process opens the same environment multiple times, closing it once will
|
||||
remove all the locks held on it, and the other instances will be
|
||||
vulnerable to corruption from other processes.
|
||||
|
||||
For compatibility with LMDB which allows multi-opening, MDBX can be
|
||||
configured at runtime by `mdbx_setup_debug(MDBX_DBG_LEGACY_MULTIOPEN, ...)`
|
||||
prior to calling other MDBX functions. In this way MDBX will track
|
||||
databases opening, detect multi-opening cases and then recover POSIX file
|
||||
locks as necessary. However, lock recovery can cause unexpected pauses,
|
||||
such as when another process opened the database in exclusive mode before
|
||||
the lock was restored - we have to wait until such a process releases the
|
||||
database, and so on.
|
||||
|
||||
|
||||
## Long-lived read transactions {#long-lived-read}
|
||||
Avoid long-lived read transactions, especially in the scenarios with a
|
||||
@ -162,6 +50,7 @@ The "next" version of libmdbx (\ref MithrilDB) will completely solve this.
|
||||
not apply to write transactions if the system clears stale writers, see
|
||||
above.
|
||||
|
||||
|
||||
## Large data items and huge transactions
|
||||
|
||||
MDBX allows you to store values up to 1 gigabyte in size, but this is
|
||||
@ -191,10 +80,123 @@ list of pages to be retired.
|
||||
|
||||
Both of these issues will be addressed in MithrilDB.
|
||||
|
||||
|
||||
## Space reservation
|
||||
An MDBX database configuration will often reserve considerable unused
|
||||
memory address space and maybe file size for future growth. This does
|
||||
not use actual memory or disk space, but users may need to understand
|
||||
the difference so they won't be scared off.
|
||||
|
||||
\todo To write about the Read/Write Amplification Factors
|
||||
|
||||
## Remote filesystems
|
||||
Do not use MDBX databases on remote filesystems, even between processes
|
||||
on the same host. This breaks file locks on some platforms, possibly
|
||||
memory map sync, and certainly sync between programs on different hosts.
|
||||
|
||||
On the other hand, MDBX support the exclusive database operation over
|
||||
a network, and cooperative read-only access to the database placed on
|
||||
a read-only network shares.
|
||||
|
||||
|
||||
## Child processes
|
||||
Do not use opened \ref MDBX_env instance(s) in a child processes after `fork()`.
|
||||
It would be insane to call fork() and any MDBX-functions simultaneously
|
||||
from multiple threads. The best way is to prevent the presence of open
|
||||
MDBX-instances during `fork()`.
|
||||
|
||||
The \ref MDBX_ENV_CHECKPID build-time option, which is ON by default on
|
||||
non-Windows platforms (i.e. where `fork()` is available), enables PID
|
||||
checking at a few critical points. But this does not give any guarantees,
|
||||
but only allows you to detect such errors a little sooner. Depending on
|
||||
the platform, you should expect an application crash and/or database
|
||||
corruption in such cases.
|
||||
|
||||
On the other hand, MDBX allow calling \ref mdbx_env_close() in such cases to
|
||||
release resources, but no more and in general this is a wrong way.
|
||||
|
||||
|
||||
## Read-only mode
|
||||
There is no pure read-only mode in a normal explicitly way, since
|
||||
readers need write access to LCK-file to be ones visible for writer.
|
||||
|
||||
So MDBX always tries to open/create LCK-file for read-write, but switches
|
||||
to without-LCK mode on appropriate errors (`EROFS`, `EACCESS`, `EPERM`)
|
||||
if the read-only mode was requested by the \ref MDBX_RDONLY flag which is
|
||||
described below.
|
||||
|
||||
The "next" version of libmdbx (\ref MithrilDB) will solve this issue for the "many
|
||||
readers without writer" case.
|
||||
|
||||
|
||||
## Troubleshooting the LCK-file
|
||||
1. A broken LCK-file can cause sync issues, including appearance of
|
||||
wrong/inconsistent data for readers. When database opened in the
|
||||
cooperative read-write mode the LCK-file requires to be mapped to
|
||||
memory in read-write access. In this case it is always possible for
|
||||
stray/malfunctioned application could writes thru pointers to
|
||||
silently corrupt the LCK-file.
|
||||
|
||||
Unfortunately, there is no any portable way to prevent such
|
||||
corruption, since the LCK-file is updated concurrently by
|
||||
multiple processes in a lock-free manner and any locking is
|
||||
unwise due to a large overhead.
|
||||
|
||||
The "next" version of libmdbx (\ref MithrilDB) will solve this issue.
|
||||
|
||||
\note Workaround: Just make all programs using the database close it;
|
||||
the LCK-file is always reset on first open.
|
||||
|
||||
2. Stale reader transactions left behind by an aborted program cause
|
||||
further writes to grow the database quickly, and stale locks can
|
||||
block further operation.
|
||||
MDBX checks for stale readers while opening environment and before
|
||||
growth the database. But in some cases, this may not be enough.
|
||||
|
||||
\note Workaround: Check for stale readers periodically, using the
|
||||
\ref mdbx_reader_check() function or the mdbx_stat tool.
|
||||
|
||||
3. Stale writers will be cleared automatically by MDBX on supported
|
||||
platforms. But this is platform-specific, especially of
|
||||
implementation of shared POSIX-mutexes and support for robust
|
||||
mutexes. For instance there are no known issues on Linux, OSX,
|
||||
Windows and FreeBSD.
|
||||
|
||||
\note Workaround: Otherwise just make all programs using the database
|
||||
close it; the LCK-file is always reset on first open of the environment.
|
||||
|
||||
|
||||
## One thread - One transaction
|
||||
A thread can only use one transaction at a time, plus any nested
|
||||
read-write transactions in the non-writemap mode. Each transaction
|
||||
belongs to one thread. The \ref MDBX_NOTLS flag changes this for read-only
|
||||
transactions. See below.
|
||||
|
||||
Do not start more than one transaction for a one thread. If you think
|
||||
about this, it's really strange to do something with two data snapshots
|
||||
at once, which may be different. MDBX checks and preventing this by
|
||||
returning corresponding error code (\ref MDBX_TXN_OVERLAPPING, \ref MDBX_BAD_RSLOT,
|
||||
\ref MDBX_BUSY) unless you using \ref MDBX_NOTLS option on the environment.
|
||||
Nonetheless, with the `MDBX_NOTLS` option, you must know exactly what you
|
||||
are doing, otherwise you will get deadlocks or reading an alien data.
|
||||
|
||||
|
||||
## Do not open twice
|
||||
Do not have open an MDBX database twice in the same process at the same
|
||||
time. By default MDBX prevent this in most cases by tracking databases
|
||||
opening and return \ref MDBX_BUSY if anyone LCK-file is already open.
|
||||
|
||||
The reason for this is that when the "Open file description" locks (aka
|
||||
OFD-locks) are not available, MDBX uses POSIX locks on files, and these
|
||||
locks have issues if one process opens a file multiple times. If a single
|
||||
process opens the same environment multiple times, closing it once will
|
||||
remove all the locks held on it, and the other instances will be
|
||||
vulnerable to corruption from other processes.
|
||||
|
||||
For compatibility with LMDB which allows multi-opening, MDBX can be
|
||||
configured at runtime by `mdbx_setup_debug(MDBX_DBG_LEGACY_MULTIOPEN, ...)`
|
||||
prior to calling other MDBX functions. In this way MDBX will track
|
||||
databases opening, detect multi-opening cases and then recover POSIX file
|
||||
locks as necessary. However, lock recovery can cause unexpected pauses,
|
||||
such as when another process opened the database in exclusive mode before
|
||||
the lock was restored - we have to wait until such a process releases the
|
||||
database, and so on.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user