77 KiB
ChangeLog
English version by Google and by Yandex.
v0.12.3 (Акула) запланирован на 2022-12-11
Выпуск с существенными доработками и новой функциональностью в память о закрытом open-source проекте "Акула".
18 files changed, 2729 insertions(+), 1696 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
Благодарности:
- Alex Sharov и команде Erigon за тестирование.
- Simon Leier за сообщение о сбоях и тестирование.
Новое:
-
Использование адреса https://libmdbx.dqdkfa.ru/dead-github для отсылки к сохранённым в web.archive.org копиям ресурсов, уничтоженных администрацией Github.
-
Реализована prefault-запись при выделении страниц для read-write отображений. Это приводит к кратному снижению системных издержек и существенному увеличению производительности в соответствующих сценариях использования, когда:
- размер БД и объём данных существенно больше ОЗУ;
- используется режим
MDBX_WRITEMAP
; - не-мелкие транзакции (по ходу транзакции выделяется многие сотни или тысячи страниц).
В режиме
MDBX_WRITEMAP
выделение/переиспользование страниц приводит к page-fault и чтению страницы с диска, даже если содержимое страницы не нужно (будет перезаписано). Это является следствием работы подсистемы виртуальной памяти, а штатный способ лечения черезMADV_REMOVE
работает не на всех ФС и обычно дороже получаемой экономии.Теперь в libmdbx используется "упреждающая запись" таких страниц, которая на системах с unified page cache приводит к "вталкиванию" данных, устраняя необходимость чтения с диска при обращении к такой странице памяти.
Новый функционал работает в согласованности с автоматическим управлением read-ahead и кэшем статуса присутствия страниц в ОЗУ, посредством mincore().
-
Добавлена опция
MDBX_opt_prefault_write_enable
для возможности принудительного включения/выключения prefault-записи. -
Реализован динамический выбор между сквозной записью на диск и обычной записью с последующим fdatasync() управляемый опцией
MDBX_opt_writethrough_threshold
.В долговечных (durable) режимах данные на диск могут быть сброшены двумя способами:
- сквозной записью через файловый дескриптор открытый с
O_DSYNC
; - обычной записью с последующим вызовом
fdatasync()
.
Первый способ выгоднее при записи малого количества страниц и/или если канал взаимодействия с диском/носителем имеет близкую к нулю задержку. Второй способ выгоднее если требуется записать много страниц и/или канал взаимодействия имеет весомую задержку (датацентры, облака). Добавленная опция
MDBX_opt_writethrough_threshold
позволяет во время выполнения задать порог для динамического выбора способа записи в зависимост от объема и конкретных условия использования. - сквозной записью через файловый дескриптор открытый с
-
Автоматическая установка
MDBX_opt_rp_augment_limit
в зависимости от размера БД. -
Запрещение разного режима
MDBX_WRITEMAP
между процессами в режимах с отложенной/ленивой записью, так как в этом случае невозможно обеспечить сброс данных на диск во всех случаях на всех поддерживаемых платформах.
Исправления (без корректировок новых функций):
-
Изменение размера отображения если это требуется для сброса данных на диск при вызове
mdbx_env_sync()
из параллельного потока выполнения вне работающей транзакции. -
Исправление регресса после коммита
db72763de0
от 2022-10-08 в логике возврата грязных страниц в режимеMDBX_WRITEMAP
, из-за чего освободившиеся страницы использовались не немедленно, а попадали в retired-список совершаемой транзакции и происходил необоснованный рост размера транзакции. -
Устранение SIGSEGV или ошибочного вызова
free()
в ситуациях повторного открытия среды посредствомmdbx_env_open()
. -
Устранение ошибки совершенной в коммите
fe20de136c
от 2022-09-18, в результате чего на Linux в режимеMDBX_WRITEMAP
никогда не вызывалсяmsync()
. Проблема существует только в релизе 0.12.2. -
Добавление подсчета грязных страниц в
MDBX_WRITEMAP
для предоставления посредствомmdbx_txn_info()
актуальной информации об объеме изменений в процессе транзакций чтения-записи. -
Исправление несущественной опечатки в условиях
#if
определения порядка байт. -
Исправление сборки для случая
MDBX_PNL_ASCENDING=1
.
Ликвидация технических долгов и мелочи:
- Доработка поддержки авто-слияния записей GC внутри
page_alloc_slowpath()
. - Устранение несущественных предупреждений Coverity.
- Использование единого курсора для поиска в GC.
- Переработка внутренних флагов связанных с выделением страниц из GC.
- Доработка подготовки резерва перед обновлением GC при включенном BigFoot.
- Оптимизация
pnl_merge()
для случаев неперекрывающихся объединяемых списков. - Оптимизация поддержки отсортированного списка страниц в
dpl_append()
. - Ускорение работы
mdbx_chk
при обработке пользовательских записей в@MAIN
.
v0.12.2 (Иван Ярыгин) от 2022-11-11
Выпуск с существенными доработками и новой функциональностью в память о российском борце Иване Сергеевиче Ярыгине.
На Олимпийских играх в Мюнхене в 1972 году Иван Ярыгин уложил всех соперников на лопатки, суммарно затратив менее 9 минут. Этот рекорд никем не побит до сих пор.
64 files changed, 5573 insertions(+), 2510 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
Новое:
-
Поддержка всех основных опций при сборке посредством CMake.
-
Требования к CMake понижены до версии 3.0.2 для возможности сборки для устаревших платформ.
-
Добавлена возможность профилирования работы GC в сложных и/или нагруженных сценариях (например Ethereum/Erigon). По-умолчанию соответствующий код отключен, а для его активации необходимо указать опцию сборки
MDBX_ENABLE_PROFGC=1
. -
Добавлена функция
mdbx_env_warmup()
для "прогрева" БД с возможностью закрепления страниц в памяти. В утилитыmdbx_chk
,mdbx_copy
иmdbx_dump
добавлены опции-u
и-U
для активации соответствующего функционала. -
Отключение учета «грязных» страниц в не требующих этого режимах (
MDBX_WRITEMAP
приMDBX_AVOID_MSYNC=0
). Доработка позволяет снизить накладные расходы и была запланирована давно, но откладывалась так как требовала других изменений. -
Вытеснение из памяти (спиллинг) «грязных» страниц с учетом размера large/overflow-страниц. Доработка позволяет корректно соблюдать политику задаваемую опциями
MDBX_opt_txn_dp_limit
,MDBX_opt_spill_max_denominator
,MDBX_opt_spill_min_denominator
и была запланирована давно, но откладывалась так как требовала других изменений. -
Для Windows в API добавлены UNICODE-зависимые определения макросов
MDBX_DATANAME
,MDBX_LOCKNAME
иMDBX_LOCK_SUFFIX
. -
Переход на преимущественное использование типа
size_t
для уменьшения накладных расходов на платформе Эльбрус. -
В API добавлены функции
mdbx_limits_valsize4page_max()
иmdbx_env_get_valsize4page_max()
возвращающие максимальный размер в байтах значения, которое может быть размещена в одной large/overflow-странице, а не последовательности из двух или более таких страниц. Для таблиц с поддержкой дубликатов вынос значений на large/overflow-страницы не поддерживается, поэтому результат совпадает сmdbx_limits_valsize_max()
. -
В API добавлены функции
mdbx_limits_pairsize4page_max()
иmdbx_env_get_pairsize4page_max()
возвращающие в байтах максимальный суммарный размер пары ключ-значение для их размещения на одной листовой страницы, без выноса значения на отдельную large/overflow-страницу. Для таблиц с поддержкой дубликатов вынос значений на large/overflow-страницы не поддерживается, поэтому результат определяет максимальный/допустимый суммарный размер пары ключ-значение. -
Реализовано использование асинхронной (overlapped) записи в Windows, включая использования небуфферизированного ввода-вывода и
WriteGather()
. Это позволяет сократить накладные расходы и частично обойти проблемы Windows с низкой производительностью ввода-вывода, включая большие задержкиFlushFileBuffers()
. Новый код также обеспечивает консолидацию записываемых регионов на всех платформах, а на Windows использование событий (events) сведено к минимум, одновременно с автоматических использованиемWriteGather()
. Поэтому ожидается существенное снижение накладных расходов взаимодействия с ОС, а в Windows это ускорение, в некоторых сценариях, может быть кратным в сравнении с LMDB. -
Добавлена опция сборки
MDBX_AVOID_MSYNC
, которая определяет поведение libmdbx в режимеMDBX_WRITE_MAP
(когда данные изменяются непосредственно в отображенных в ОЗУ страницах БД):-
Если
MDBX_AVOID_MSYNC=0
(по умолчанию на всех системах кроме Windows), то (как прежде) сохранение данных выполняется посредствомmsync()
, либоFlushViewOfFile()
на Windows. На платформах с полноценной подсистемой виртуальной памяти и адекватным файловым вводом-выводом это обеспечивает минимум накладных расходов (один системный вызов) и максимальную производительность. Однако, на Windows приводит к значительной деградации, в том числе из-за того что послеFlushViewOfFile()
требуется также вызовFlushFileBuffers()
с массой проблем и суеты внутри ядра ОС. -
Если
MDBX_AVOID_MSYNC=1
(по умолчанию только на Windows), то сохранение данных выполняется явной записью в файл каждой измененной страницы БД. Это требует дополнительных накладных расходов, как на отслеживание измененных страниц (ведение списков "грязных" страниц), так и на системные вызовы для их записи. Кроме этого, с точки зрения подсистемы виртуальной памяти ядра ОС, страницы БД измененные в ОЗУ и явно записанные в файл, могут либо оставаться "грязными" и быть повторно записаны ядром ОС позже, либо требовать дополнительных накладных расходов для отслеживания PTE (Page Table Entries), их модификации и дополнительного копирования данных. Тем не менее, по имеющейся информации, на Windows такой путь записи данных в целом обеспечивает более высокую производительность.
-
-
Улучшение эвристики включения авто-слияния записей GC.
-
Изменение формата LCK и семантики некоторых внутренних полей. Версии libmdbx использующие разный формат не смогут работать с одной БД одновременно, а только поочередно (LCK-файл переписывается при открытии первым открывающим БД процессом).
-
В
C++
API добавлены методы фиксации транзакции с получением информации о задержках. -
Added
MDBX_HAVE_BUILT IN_CPU_SUPPORTS
build option to control use GCC's__builtin_cpu_supports()
function, which could be unavailable on a fake OSes (macos, ios, android, etc).
Исправления (без корректировок вышеперечисленных новых функций):
- Устранения ряда предупреждений при сборке посредством MinGW.
- Устранение ложно-положительных сообщений от Valgrind об использовании
не инициализированных данных из-за выравнивающих зазоров в
struct troika
. - Исправлен возврат неожиданной ошибки
MDBX_BUSY
из функцийmdbx_env_set_option()
,mdbx_env_set_syncbytes()
иmdbx_env_set_syncperiod()
. - Небольшие исправления для совместимости с CMake 3.8
- Больше контроля и осторожности (паранойи) для страховки от дефектов
mremap()
. - Костыль для починки сборки со старыми версиями
stdatomic.h
из GNU Lib C, где макросыATOMIC_*_LOCK_FREE
ошибочно переопределяются через функции. - Использование
fcntl64(F_GETLK64/F_SETLK64/F_SETLKW64)
при наличии. Это решает проблему срабатывания проверочного утверждения при сборке для платформ где типoff_t
шире соответствующих полейструктуры flock
, используемой для блокировки файлов. - Доработан сбор информации о задержках при фиксации транзакций:
- Устранено искажение замеров длительности обновления GC при включении отладочного внутреннего аудита;
- Защита от undeflow-нуля только общей задержки в метриках, чтобы исключить ситуации, когда сумма отдельных стадий больше общей длительности.
- Ряд исправлений для устранения срабатываний проверочных утверждения в отладочных сборках.
- Более осторожное преобразование к типу
mdbx_tid_t
для устранения предупреждений. - Исправление лишнего сброса данных на диск в режиме
MDBX_SAFE_NOSYNC
при обновлении GC. - Fixed an extra check for
MDBX_APPENDDUP
insidemdbx_cursor_put()
which could result in returningMDBX_EKEYMISMATCH
for valid cases. - Fixed nasty
clz()
bug (by using_BitScanReverse()
, only MSVC builds affected).
Мелочи:
- Исторические ссылки cвязанные с удалённым на
githubпроектом перенаправлены на web.archive.org. - Синхронизированны конструкции CMake между проектами.
- Добавлено предупреждение о небезопасности RISC-V.
- Добавлено описание параметров
MDBX_debug_func
иMDBX_debug_func
. - Добавлено обходное решение для минимизации ложно-положительных конфликтов при использовании файловых блокировок в Windows.
- Проверка атомарности C11-операций c 32/64-битными данными.
- Уменьшение в 42 раза значения по-умолчанию для
me_options.dp_limit
в отладочных сборках. - Добавление платформы
gcc-riscv64-linux-gnu
в список для целиcross-gcc
. - Небольшие правки скрипта
long_stochastic.sh
для работы в Windows. - Удаление ненужного вызова
LockFileEx()
внутриmdbx_env_copy()
. - Добавлено описание использования файловых дескрипторов в различных режимах.
- Добавлено использование
_CrtDbgReport()
в отладочных сборках. - Fixed an extra ensure/assertion check of
oldest_reader
insidetxn_end()
. - Removed description of deprecated usage of
MDBX_NODUPDATA
. - Fixed regression ASAN/Valgring-enabled builds.
- Fixed minor MingGW warning.
v0.12.1 (Positive Proxima) at 2022-08-24
The planned frontward release with new superior features on the day of 20 anniversary of Positive Technologies.
37 files changed, 7604 insertions(+), 7417 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
New:
- The
Big Foot
feature which significantly reduces GC overhead for processing large lists of retired pages from huge transactions. Now libmdbx avoid creating large chunks of PNLs (page number lists) which required a long sequences of free pages, aka large/overflow pages. Thus avoiding searching, allocating and storing such sequences inside GC. - Improved hot/online validation and checking of database pages both for more robustness and performance.
- New solid and fast method to latch meta-pages called
Troika
. The minimum of memory barriers, reads, comparisons and conditional transitions are used. - New
MDBX_VALIDATION
environment options to extra validation of DB structure and pages content for carefully/safe handling damaged or untrusted DB. - Accelerated ×16/×8/×4 by AVX512/AVX2/SSE2/Neon implementations of search page sequences.
- Added the
gcrtime_seconds16dot16
counter to the "Page Operation Statistics" that accumulates time spent for GC searching and reclaiming. - Copy-with-compactification now clears/zeroes unused gaps inside database pages.
- The
C
andC++
APIs has been extended and/or refined to simplify usingwchar_t
pathnames. On Windows themdbx_env_openW()
, ``mdbx_env_get_pathW()()
,mdbx_env_copyW()
,mdbx_env_open_for_recoveryW()
are available for now, but themdbx_env_get_path()
has been replaced in favor ofmdbx_env_get_pathW()
. - Added explicit error message for Buildroot's Microblaze toolchain maintainers.
- Added
MDBX_MANAGE_BUILD_FLAGS
build options for CMake. - Speed-up internal
bsearch
/lower_bound
implementation using branchless tactic, including workaround for CLANG x86 optimiser bug. - A lot internal refinement and micro-optimisations.
- Internally counted volume of dirty pages (unused for now but for coming features).
Fixes:
- Never use modern
__cxa_thread_atexit()
on Apple's OSes. - Don't check owner for finished transactions.
- Fixed typo in
MDBX_EINVAL
which breaks MingGW builds with CLANG.
v0.12.0 at 2022-06-19
Not a release but preparation for changing feature set and API.
v0.11.13 at (Swashplate) 2022-11-10
The stable bugfix release in memory of Boris Yuryev on his 133rd birthday.
30 files changed, 405 insertions(+), 136 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
Fixes:
- Fixed builds with older libc versions after using
fcntl64()
(backport). - Fixed builds with older
stdatomic.h
versions, where theATOMIC_*_LOCK_FREE
macros mistakenly redefined using functions (backport). - Added workaround for
mremap()
defect to avoid assertion failure (backport). - Workaround for
encryptfs
bug(s) in thecopy_file_range
implementation (backport). - Fixed unexpected
MDBX_BUSY
frommdbx_env_set_option()
,mdbx_env_set_syncbytes()
andmdbx_env_set_syncperiod()
(backport). - CMake requirements lowered to version 3.0.2 (backport).
Minors:
- Minor clarification output of
--help
formdbx_test
(backport). - Added admonition of insecure for RISC-V (backport).
- Stochastic scripts and CMake files synchronized with the
devel
branch. - Use
--dont-check-ram-size
for small-tests make-targets (backport).
v0.11.12 (Эребуни) at 2022-10-12
The stable bugfix release.
11 files changed, 96 insertions(+), 49 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
Fixes:
- Fixed static assertion failure on platforms where the
off_t
type is wider than corresponding fields ofstruct flock
used for file locking (backport). Now libmdbx will usefcntl64(F_GETLK64/F_SETLK64/F_SETLKW64)
if available. - Fixed assertion check inside
page_retire_ex()
(backport).
Minors:
- Fixed
-Wint-to-pointer-cast
warnings while casting tomdbx_tid_t
(backport). - Removed needless
LockFileEx()
insidemdbx_env_copy()
(backport).
v0.11.11 (Тендра-1790) at 2022-09-11
The stable bugfix release.
10 files changed, 38 insertions(+), 21 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
Fixes:
- Fixed an extra check for
MDBX_APPENDDUP
insidemdbx_cursor_put()
which could result in returningMDBX_EKEYMISMATCH
for valid cases. - Fixed an extra ensure/assertion check of
oldest_reader
insidemdbx_txn_end()
. - Fixed derived C++ builds by removing
MDBX_INTERNAL_FUNC
formdbx_w2mb()
andmdbx_mb2w()
.
v0.11.10 (the TriColor) at 2022-08-22
The stable bugfix release.
14 files changed, 263 insertions(+), 252 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
New:
- The C++ API has been refined to simplify support for
wchar_t
in path names. - Added explicit error message for Buildroot's Microblaze toolchain maintainers.
Fixes:
- Never use modern
__cxa_thread_atexit()
on Apple's OSes. - Use
MultiByteToWideChar(CP_THREAD_ACP)
instead ofmbstowcs()
. - Don't check owner for finished transactions.
- Fixed typo in
MDBX_EINVAL
which breaks MingGW builds with CLANG.
Minors:
- Fixed variable name typo.
- Using
ldd
to check used dso. - Added
MDBX_WEAK_IMPORT_ATTRIBUTE
macro. - Use current transaction geometry for untouched parameters when
env_set_geometry()
called within a write transaction. - Minor clarified
iov_page()
failure case.
v0.11.9 (Чирчик-1992) at 2022-08-02
The stable bugfix release.
18 files changed, 318 insertions(+), 178 deletions(-)
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
Acknowledgements:
- Alex Sharov and Erigon team for reporting and testing.
- Andrew Ashikhmin for contributing.
New:
- Ability to customise
MDBX_LOCK_SUFFIX
,MDBX_DATANAME
,MDBX_LOCKNAME
just by predefine ones during build. - Added to
mdbx::env_managed
's methods a few overloads withconst char* pathname
parameter (C++ API).
Fixes:
- Fixed hang copy-with-compactification of a corrupted DB
or in case the volume of output pages is a multiple of
MDBX_ENVCOPY_WRITEBUF
. - Fixed standalone non-CMake build on MacOS (
#include AvailabilityMacros.h>
). - Fixed unexpected
MDBX_PAGE_FULL
error in rare cases with large database page sizes.
Minors:
- Minor fixes Doxygen references, comments, descriptions, etc.
- Fixed copy&paste typo inside
meta_checktxnid()
. - Minor fix
meta_checktxnid()
to avoid assertion in debug mode. - Minor fix
mdbx_env_set_geometry()
to avoid returningEINVAL
in particular rare cases. - Minor refine/fix batch-get testcase for large page size.
- Added
--pagesize NN
option to long-stotastic test script. - Updated Valgrind-suppressions file for modern GCC.
- Fixed
has no symbols
warning from Apple's ranlib.
v0.11.8 (Baked Apple) at 2022-06-12
The stable release with an important fixes and workaround for the critical macOS thread-local-storage issue.
Acknowledgements:
New:
- Added most of transactions flags to the public API.
- Added
MDBX_NOSUCCESS_EMPTY_COMMIT
build option to return non-success result (MDBX_RESULT_TRUE
) on empty commit. - Reworked validation and import of DBI-handles into a transaction. Assumes these changes will be invisible to most users, but will cause fewer surprises in complex DBI cases.
- Added ability to open DB in without-LCK (exclusive read-only) mode in case no permissions to create/write LCK-file.
Fixes:
- A series of fixes and improvements for automatically generated documentation (Doxygen).
- Fixed copy&paste bug with could lead to
SIGSEGV
(nullptr dereference) in the exclusive/no-lck mode. - Fixed minor warnings from modern Apple's CLANG 13.
- Fixed minor warnings from CLANG 14 and in-development CLANG 15.
- Fixed
SIGSEGV
regression in without-LCK (exclusive read-only) mode. - Fixed
mdbx_check_fs_local()
for CDROM case on Windows. - Fixed nasty typo of typename which caused false
MDBX_CORRUPTED
error in a rare execution path, when the size of the thread-ID type not equal to 8. - Fixed Elbrus/E2K LCC 1.26 compiler warnings (memory model for atomic operations, etc).
- Fixed write-after-free memory corruption on latest
macOS
during finalization/cleanup of thread(s) that executed read transaction(s).The issue was suddenly discovered by a CI after adding an iteration with macOS 11 "Big Sur", and then reproduced on recent release of macOS 12 "Monterey". The issue was never noticed nor reported on macOS 10 "Catalina" nor others. Analysis shown that the problem caused by a change in the behavior of the system library (internals of dyld and pthread) during thread finalization/cleanup: now a memory allocated for a
__thread
variable(s) is released before execution of the registered Thread-Local-Storage destructor(s), thus a TLS-destructor will write-after-free just by legitime dereference any__thread
variable. This is unexpected crazy-like behavior since the order of resources releasing/destroying is not the reverse of ones acquiring/construction order. Nonetheless such surprise is now workarounded by using atomic compare-and-swap operations on a 64-bit signatures/cookies.
Minors:
- Refined
release-assets
GNU Make target. - Added logging to
mdbx_fetch_sdb()
to help debugging complex DBI-handels use cases. - Added explicit error message from probe of no-support for
std::filesystem
. - Added contributors "score" table by
git fame
to generated docs. - Added
mdbx_assert_fail()
to public API (mostly for backtracing). - Now C++20 concepts used/enabled only when
__cpp_lib_concepts >= 202002
. - Don't provide nor report package information if used as a CMake subproject.
v0.11.7 (Resurrected Sarmat) at 2022-04-22
The stable risen release after the Github's intentional malicious disaster.
We have migrated to a reliable trusted infrastructure
The origin for now is at GitFlic
since on 2022-04-15 the Github administration, without any warning nor
explanation, deleted libmdbx along with a lot of other projects,
simultaneously blocking access for many developers.
For the same reason Github is blacklisted forever.
GitFlic already support Russian and English languages, plan to support more, including 和 中文. You are welcome!
New:
- Added the
tools-static
make target to build statically linked MDBX tools. - Support for Microsoft Visual Studio 2022.
- Support build by MinGW' make from command line without CMake.
- Added
mdbx::filesystem
C++ API namespace that corresponds tostd::filesystem
orstd::experimental::filesystem
. - Created website for online auto-generated documentation.
- Used
https://web.archive.org/web/https://github.com/erthink/libmdbx
for dead (or temporarily lost) resources deleted byGithub. - Added
--loglevel=
command-line option to themdbx_test
tool. - Added few fast smoke-like tests into CMake builds.
Fixes:
- Fixed a race between starting a transaction and creating a DBI descriptor that could lead to
SIGSEGV
in the cursor tracking code. - Clarified description of
MDBX_EPERM
error returned frommdbx_env_set_geometry()
. - Fixed non-promoting the parent transaction to be dirty in case the undo of the geometry update failed during abortion of a nested transaction.
- Resolved linking issues with
libstdc++fs
/libc++fs
/libc++experimental
for C++std::filesystem
orstd::experimental::filesystem
for legacy compilers. - Added workaround for GNU Make 3.81 and earlier.
- Added workaround for Elbrus/LCC 1.25 compiler bug of class inline
static constexpr
member field. - Fixed minor assertion regression (only debug builds were affected).
- Fixed detection of
C++20
concepts accessibility. - Fixed detection of Clang's LTO availability for Android.
- Fixed extra definition of
_FILE_OFFSET_BITS=64
for Android that is problematic for 32-bit Bionic. - Fixed build for ARM/ARM64 by MSVC.
- Fixed non-x86 Windows builds with
MDBX_WITHOUT_MSVC_CRT=ON
andMDBX_BUILD_SHARED_LIBRARY=ON
.
Minors:
- Resolve minor MSVC warnings: avoid
/INCREMENTAL[:YES]
with/LTCG
,/W4
with/W3
, theC5105
warning. - Switched to using
MDBX_EPERM
instead ofMDBX_RESULT_TRUE
to indicate that the geometry cannot be updated. - Added
NULL
checking during memory allocation insidemdbx_chk
. - Resolved all warnings from MinGW while used without CMake.
- Added inheretable
target_include_directories()
toCMakeLists.txt
for easy integration. - Added build-time checks and paranoid runtime assertions for the
off_t
arguments offcntl()
which are used for locking. - Added
-Wno-lto-type-mismatch
to avoid false-positive warnings from old GCC during LTO-enabled builds. - Added checking for TID (system thread id) to avoid hang on 32-bit Bionic/Android within
pthread_mutex_lock()
. - Reworked
MDBX_BUILD_TARGET
of CMake builds. - Added
CMAKE_HOST_ARCH
andCMAKE_HOST_CAN_RUN_EXECUTABLES_BUILT_FOR_TARGET
.
v0.11.6 at 2022-03-24
The stable release with the complete workaround for an incoherence flaw of Linux unified page/buffer cache. Nonetheless the cause for this trouble may be an issue of Intel CPU cache/MESI. See issue#269 for more information.
Acknowledgements:
- David Bouyssié for Scala bindings.
- Michelangelo Riccobene for reporting and testing.
Fixes:
- Added complete workaround for an incoherence flaw of Linux unified page/buffer cache.
- Fixed cursor reusing for read-only transactions.
- Fixed copy&paste typo inside
mdbx::cursor::find_multivalue()
.
Minors:
- Minor refine C++ API for convenience.
- Minor internals refines.
- Added
lib-static
andlib-shared
targets for make. - Added minor workaround for AppleClang 13.3 bug.
- Clarified error messages of a signature/version mismatch.
v0.11.5 at 2022-02-23
The release with the temporary hotfix for a flaw of Linux unified page/buffer cache. See issue#269 for more information.
Acknowledgements:
- Simon Leier for reporting and testing.
- Kai Wetlesen for RPMs.
- Tullio Canepa for reporting C++ API issue and contributing.
Fixes:
- Added hotfix for a flaw of Linux unified page/buffer cache.
- Fixed/Reworked move-assignment operators for "managed" classes of C++ API.
- Fixed potential
SIGSEGV
while open DB with overrided non-default page size. - Made
mdbx_env_open()
idempotence in failure cases. - Refined/Fixed pages reservation inside
mdbx_update_gc()
to avoid non-reclamation in a rare cases. - Fixed typo in a retained space calculation for the hsr-callback.
Minors:
- Reworked functions for meta-pages, split-off non-volatile.
- Disentangled C11-atomic fences/barriers and pure-functions (with
__attribute__((__pure__))
) to avoid compiler misoptimization. - Fixed hypotetic unaligned access to 64-bit dwords on ARM with
__ARM_FEATURE_UNALIGNED
defined. - Reasonable paranoia that makes clarity for code readers.
- Minor fixes Doxygen references, comments, descriptions, etc.
v0.11.4 at 2022-02-02
The stable release with fixes for large and huge databases sized of 4..128 TiB.
Acknowledgements:
- Ledgerwatch, Binance and Positive Technologies teams for reporting, assistance in investigation and testing.
- Alex Sharov for reporting, testing and provide resources for remote debugging/investigation.
- Kris Zyp for Deno support.
New features, extensions and improvements:
- Added treating the
UINT64_MAX
value as maximum for given option insidemdbx_env_set_option()
. - Added
to_hex/to_base58/to_base64::output(std::ostream&)
overloads without using temporary string objects as buffers. - Added
--geometry-jitter=YES|no
option to the test framework. - Added support for Deno support by Kris Zyp.
Fixes:
- Fixed handling
MDBX_opt_rp_augment_limit
for GC's records from huge transactions (Erigon/Akula/Ethereum). - Fixed build on Android (avoid including
sys/sem.h
). - Fixed missing copy assignment operator for
mdbx::move_result
. - Fixed missing
&
forstd::ostream &operator<<()
overloads. - Fixed unexpected
EXDEV
(Cross-device link) error frommdbx_env_copy()
. - Fixed base64 encoding/decoding bugs in auxillary C++ API.
- Fixed overflow of
pgno_t
during checking PNL on 64-bit platforms. - Fixed excessive PNL checking after sort for spilling.
- Reworked checking
MAX_PAGENO
and DB upper-size geometry limit. - Fixed build for some combinations of versions of MSVC and Windows SDK.
Minors:
- Added workaround for CLANG bug D79919/PR42445.
- Fixed build test on Android (using
pthread_barrier_t
stub). - Disabled C++20 concepts for CLANG < 14 on Android.
- Fixed minor
unused parameter
warning. - Added CI for Android.
- Refine/cleanup internal logging.
- Refined line splitting inside hex/base58/base64 encoding to avoid
\n
at the end. - Added workaround for modern libstdc++ with CLANG < 4.x
- Relaxed txn-check rules for auxiliary functions.
- Clarified a comments and descriptions, etc.
- Using the
-fno-semantic interposition
option to reduce the overhead to calling self own public functions.
v0.11.3 at 2021-12-31
Acknowledgements:
- gcxfd i@rmw.link for reporting, contributing and testing.
- 장세연 (Чан Се Ен) for reporting and testing.
- Alex Sharov for reporting, testing and provide resources for remote debugging/investigation.
New features, extensions and improvements:
- Added
mdbx_cursor_get_batch()
. - Added
MDBX_SET_UPPERBOUND
. - C++ API is finalized now.
- The GC update stage has been significantly speeded when fixing huge Erigon's transactions (Ethereum ecosystem).
Fixes:
- Disabled C++20 concepts for stupid AppleClang 13.x
- Fixed internal collision of
MDBX_SHRINK_ALLOWED
withMDBX_ACCEDE
.
Minors:
- Fixed returning
MDBX_RESULT_TRUE
(unexpected -1) frommdbx_env_set_option()
. - Added
mdbx_env_get_syncbytes()
andmdbx_env_get_syncperiod()
. - Clarified description of
MDBX_INTEGERKEY
. - Reworked/simplified
mdbx_env_sync_internal()
. - Fixed extra assertion inside
mdbx_cursor_put()
forMDBX_DUPFIXED
cases. - Avoiding extra looping inside
mdbx_env_info_ex()
. - Explicitly enabled core dumps from stochastic tests scripts on Linux.
- Fixed
mdbx_override_meta()
to avoid false-positive assertions. - For compatibility reverted returning
MDBX_ENODATA
for some cases.
v0.11.2 at 2021-12-02
Acknowledgements:
- 장세연 (Чан Се Ен) for contributing to C++ API.
- Alain Picard for Java bindings.
- Alex Sharov for reporting and testing.
- Kris Zyp for reporting and testing.
- Artem Vorotnikov for support Rust wrapper.
Fixes:
- Fixed compilation with
devtoolset-9
on CentOS/RHEL 7. - Fixed unexpected
MDBX_PROBLEM
error because of update an obsolete meta-page. - Fixed returning
MDBX_NOTFOUND
error in case an inexact value found forMDBX_GET_BOTH
operation. - Fixed compilation without kernel/libc-devel headers.
Minors:
- Fixed
constexpr
-related macros for legacy compilers. - Allowed to define 'CMAKE_CXX_STANDARD` using an environment variable.
- Simplified collection statistics of page operation .
- Added
MDBX_FORCE_BUILD_AS_MAIN_PROJECT
cmake option. - Remove unneeded
#undef P_DIRTY
.
v0.11.1 at 2021-10-23
Backward compatibility break:
The database format signature has been changed to prevent forward-interoperability with an previous releases, which may lead to a false positive diagnosis of database corruption due to flaws of an old library versions.
This change is mostly invisible:
- previously versions are unable to read/write a new DBs;
- but the new release is able to handle an old DBs and will silently upgrade ones.
Acknowledgements:
- Alex Sharov for reporting and testing.
v0.10.5 at 2021-10-13 (obsolete, please use v0.11.1)
Unfortunately, the v0.10.5
accidentally comes not full-compatible with previous releases:
v0.10.5
can read/processing DBs created by previous releases, i.e. the backward-compatibility is provided;- however, previous releases may lead to false-corrupted state with DB that was touched by
v0.10.5
, i.e. the forward-compatibility is broken forv0.10.4
and earlier.
This cannot be fixed, as it requires fixing past versions, which as a result we will just get a current version.
Therefore, it is recommended to use v0.11.1
instead of v0.10.5
.
Acknowledgements:
- Noel Kuntze for immediately bug reporting.
Fixes:
- Fixed unaligned access regression after the
#pragma pack
fix for modern compilers. - Added UBSAN-test to CI to avoid a regression(s) similar to lately fixed.
- Fixed possibility of meta-pages clashing after manually turn to a particular meta-page using
mdbx_chk
utility.
Minors:
- Refined handling of weak or invalid meta-pages while a DB opening.
- Refined providing information for the
@MAIN
and@GC
sub-databases of a last committed modification transaction's ID.
v0.10.4 at 2021-10-10
Acknowledgements:
- Artem Vorotnikov for support Rust wrapper.
- Andrew Ashikhmin for contributing to C++ API.
Fixes:
- Fixed possibility of looping update GC during transaction commit (no public issue since the problem was discovered inside Positive Technologies).
- Fixed
#pragma pack
to avoid provoking some compilers to generate code with unaligned access. - Fixed
noexcept
for potentially throwingtxn::put()
of C++ API.
Minors:
- Added stochastic test script for checking small transactions cases.
- Removed extra transaction commit/restart inside test framework.
- In debugging builds fixed a too small (single page) by default DB shrink threshold.
v0.10.3 at 2021-08-27
Acknowledgements:
- Francisco Vallarino for Haskell bindings for libmdbx.
- Alex Sharov for reporting and testing.
- Andrea Lanfranchi for contributing.
Extensions and improvements:
- Added
cursor::erase()
overloads forkey
and forkey-value
. - Resolve minor Coverity Scan issues (no fixes but some hint/comment were added).
- Resolve minor UndefinedBehaviorSanitizer issues (no fixes but some workaround were added).
Fixes:
- Always setup
madvise
while opening DB (fixes https://libmdbx.dqdkfa.ru/dead-github/issues/231). - Fixed checking legacy
P_DIRTY
flag (0x10
) for nested/sub-pages.
Minors:
- Fixed getting revision number from middle of history during amalgamation (GNU Makefile).
- Fixed search GCC tools for LTO (CMake scripts).
- Fixed/reorder dirs list for search CLANG tools for LTO (CMake scripts).
- Fixed/workarounds for CLANG < 9.x
- Fixed CMake warning about compatibility with 3.8.2
v0.10.2 at 2021-07-26
Acknowledgements:
- Alex Sharov for reporting and testing.
- Andrea Lanfranchi for reporting bugs.
- Lionel Debroux for fuzzing tests and reporting bugs.
- Sergey Fedotov for
node-mdbx
NodeJS bindings. - Kris Zyp for
lmdbx-store
NodeJS bindings. - Noel Kuntze for draft Python bindings.
New features, extensions and improvements:
- Allow to predefine/override
MDBX_BUILD_TIMESTAMP
for builds reproducibility. - Added options support for
long-stochastic
script. - Avoided
MDBX_TXN_FULL
error for large transactions when possible. - The
MDBX_READERS_LIMIT
increased to32767
. - Raise
MDBX_TOO_LARGE
under Valgrind/ASAN if being opened DB is 100 larger than RAM (to avoid hangs and OOM). - Minimized the size of poisoned/unpoisoned regions to avoid Valgrind/ASAN stuck.
- Added more workarounds for QEMU for testing builds for 32-bit platforms, Alpha and Sparc architectures.
mdbx_chk
now skips iteration & checking of DB' records if corresponding page-tree is corrupted (to avoidSIGSEGV
, ASAN failures, etc).- Added more checks for rare/fuzzing corruption cases.
Backward compatibility break:
- Use file
VERSION.txt
for version information instead ofVERSION
to avoid collision with#include <version>
. - Rename
slice::from/to_FOO_bytes()
to `slice::envisage_from/to_FOO_length()'. - Rename
MDBX_TEST_EXTRA
make's variable toMDBX_SMOKE_EXTRA
. - Some details of the C++ API have been changed for subsequent freezing.
Fixes:
- Fixed excess meta-pages checks in case
mdbx_chk
is called to check the DB for a specific meta page and thus could prevent switching to the selected meta page, even if the check passed without errors. - Fixed recursive use of SRW-lock on Windows cause by
MDBX_NOTLS
option. - Fixed log a warning during a new DB creation.
- Fixed false-negative
mdbx_cursor_eof()
result. - Fixed
make install
with non-GNUinstall
utility (OSX, BSD). - Fixed installation by
CMake
in special cases by complete useGNUInstallDirs
's variables. - Fixed C++ Buffer issue with
std::string
and alignment. - Fixed
safe64_reset()
for platforms without atomic 64-bit compare-and-swap. - Fixed hang/shutdown on big-endian platforms without
__cxa_thread_atexit()
. - Fixed using bad meta-pages if DB was partially/recoverable corrupted.
- Fixed extra
noexcept
forbuffer::&assign_reference()
. - Fixed
bootid
generation on Windows for case of change system' time. - Fixed test framework keygen-related issue.
v0.10.1 at 2021-06-01
Acknowledgements:
- Alexey Akhunov and Alex Sharov for bug reporting and testing.
- Andrea Lanfranchi for bug reporting and testing related to WSL2.
New features:
- Added
-p
option tomdbx_stat
utility for printing page operations statistic. - Added explicit checking for and warning about using unfit github's archives.
- Added fallback from OFD locking to legacy non-OFD POSIX file locks on an
EINVAL
error. - Added Plan 9 network file system to the whitelist for an ability to open a DB in exclusive mode.
- Support for opening from WSL2 environment a DB hosted on Windows drive and mounted via DrvFs (i.e by Plan 9 noted above).
Fixes:
- Fixed minor "foo not used" warnings from modern C++ compilers when building the C++ part of the library.
- Fixed confusing/messy errors when build library from unfit github's archives (https://libmdbx.dqdkfa.ru/dead-github/issues/197).
- Fixed
#elsif
typo. - Fixed rare unexpected
MDBX_PROBLEM
error during altering data in huge transactions due to wrong spilling/oust of dirty pages (https://libmdbx.dqdkfa.ru/dead-github/issues/195). - Re-Fixed WSL1/WSL2 detection with distinguishing (https://libmdbx.dqdkfa.ru/dead-github/issues/97).
v0.10.0 at 2021-05-09
Acknowledgements:
- Mahlon E. Smith for Ruby bindings.
- Alex Sharov for mdbx-go, bug reporting and testing.
- Artem Vorotnikov for bug reporting and PR.
- Paolo Rebuffo, Alexey Akhunov and Mark Grosberg for donations.
- Noel Kuntze for preliminary Python bindings
New features:
- Added
mdbx_env_set_option()
andmdbx_env_get_option()
for controls various runtime options for an environment (announce of this feature was missed in a previous news). - Added
MDBX_DISABLE_PAGECHECKS
build option to disable some checks to reduce an overhead and detection probability of database corruption to a values closer to the LMDB. TheMDBX_DISABLE_PAGECHECKS=1
provides a performance boost of about 10% in CRUD scenarios, and conjointly with theMDBX_ENV_CHECKPID=0
andMDBX_TXN_CHECKOWNER=0
options can yield up to 30% more performance compared to LMDB. - Using float point (exponential quantized) representation for internal 16-bit values of grow step and shrink threshold when huge ones (https://libmdbx.dqdkfa.ru/dead-github/issues/166). To minimize the impact on compatibility, only the odd values inside the upper half of the range (i.e. 32769..65533) are used for the new representation.
- Added the
mdbx_drop
similar to LMDB command-line tool to purge or delete (sub)database(s). - Ruby bindings is available now by Mahlon E. Smith.
- Added
MDBX_ENABLE_MADVISE
build option which controls the use of POSIXmadvise()
hints and friends. - The internal node sizes were refined, resulting in a reduction in large/overflow pages in some use cases and a slight increase in limits for a keys size to ≈½ of page size.
- Added to
mdbx_chk
output number of keys/items on pages. - Added explicit
install-strip
andinstall-no-strip
targets to theMakefile
(https://libmdbx.dqdkfa.ru/dead-github/pull/180). - Major rework page splitting (
af9b7b5605
) for- An "auto-appending" feature upon insertion for both ascending and descending key sequences. As a result, the optimality of page filling increases significantly (more densely, less slackness) while inserting ordered sequences of keys,
- A "splitting at middle" to make page tree more balanced on average.
- Added
mdbx_get_sysraminfo()
to the API. - Added guessing a reasonable maximum DB size for the default upper limit of geometry (https://libmdbx.dqdkfa.ru/dead-github/issues/183).
- Major rework internal labeling of a dirty pages (
958fd5b947
) for a "transparent spilling" feature with the gist to make a dirty pages be ready to spilling (writing to a disk) without further altering ones. Thus in theMDBX_WRITEMAP
mode the OS kernel able to oust dirty pages to DB file without further penalty during transaction commit. As a result, page swapping and I/O could be significantly reduced during extra large transactions and/or lack of memory. - Minimized reading leaf-pages during dropping subDB(s) and nested trees.
- Major rework a spilling of dirty pages to support LRU policy and prioritization for a large/overflow pages.
- Statistics of page operations (split, merge, copy, spill, etc) now available through
mdbx_env_info_ex()
. - Auto-setup limit for length of dirty pages list (
MDBX_opt_txn_dp_limit
option). - Support
make options
to list available build options. - Support
make help
to list available make targets. - Silently
make
's build by default. - Preliminary Python bindings is available now by Noel Kuntze (https://libmdbx.dqdkfa.ru/dead-github/issues/147).
Backward compatibility break:
- The
MDBX_AVOID_CRT
build option was renamed toMDBX_WITHOUT_MSVC_CRT
. This option is only relevant when building for Windows. - The
mdbx_env_stat()
always, andmdbx_env_stat_ex()
when called with the zeroed transaction parameter, now internally start temporary read transaction and thus may returnsMDBX_BAD_RSLOT
error. So, just never use deprecatedmdbx_env_stat()' and call
mdbx_env_stat_ex()` with transaction parameter. - The build option
MDBX_CONFIG_MANUAL_TLS_CALLBACK
was removed and now just a non-zero value of theMDBX_MANUAL_MODULE_HANDLER
macro indicates the requirement to manually callmdbx_module_handler()
when loading libraries and applications uses statically linked libmdbx on an obsolete Windows versions.
Fixes:
- Fixed performance regression due non-optimal C11 atomics usage (https://libmdbx.dqdkfa.ru/dead-github/issues/160).
- Fixed "reincarnation" of subDB after it deletion (https://libmdbx.dqdkfa.ru/dead-github/issues/168).
- Fixed (disallowing) implicit subDB deletion via operations on
@MAIN
's DBI-handle. - Fixed a crash of
mdbx_env_info_ex()
in case of a call for a non-open environment (https://libmdbx.dqdkfa.ru/dead-github/issues/171). - Fixed the selecting/adjustment values inside
mdbx_env_set_geometry()
for implicit out-of-range cases (https://libmdbx.dqdkfa.ru/dead-github/issues/170). - Fixed
mdbx_env_set_option()
for set initial and limit size of dirty page list ((https://libmdbx.dqdkfa.ru/dead-github/issues/179). - Fixed an unreasonably huge default upper limit for DB geometry (https://libmdbx.dqdkfa.ru/dead-github/issues/183).
- Fixed
constexpr
specifier for theslice::invalid()
. - Fixed (no)readahead auto-handling (https://libmdbx.dqdkfa.ru/dead-github/issues/164).
- Fixed non-alloy build for Windows.
- Switched to using Heap-functions instead of LocalAlloc/LocalFree on Windows.
- Fixed
mdbx_env_stat_ex()
to returning statistics of the whole environment instead of MainDB only (https://libmdbx.dqdkfa.ru/dead-github/issues/190). - Fixed building by GCC 4.8.5 (added workaround for a preprocessor's bug).
- Fixed building C++ part for iOS <= 13.0 (unavailability of
std::filesystem::path
). - Fixed building for Windows target versions prior to Windows Vista (
WIN32_WINNT < 0x0600
). - Fixed building by MinGW for Windows (https://libmdbx.dqdkfa.ru/dead-github/issues/155).
v0.9.3 at 2021-02-02
Acknowledgements:
- Mahlon E. Smith for FreeBSD port of libmdbx.
- 장세연 for bug fixing and PR.
- Clément Renault for Heed fully typed Rust wrapper.
- Alex Sharov for bug reporting.
- Noel Kuntze for bug reporting.
Removed options and features:
- Drop
MDBX_HUGE_TRANSACTIONS
build-option (now no longer required).
New features:
- Package for FreeBSD is available now by Mahlon E. Smith.
- New API functions to get/set various options (https://libmdbx.dqdkfa.ru/dead-github/issues/128):
- the maximum number of named databases for the environment;
- the maximum number of threads/reader slots;
- threshold (since the last unsteady commit) to force flush the data buffers to disk;
- relative period (since the last unsteady commit) to force flush the data buffers to disk;
- limit to grow a list of reclaimed/recycled page's numbers for finding a sequence of contiguous pages for large data items;
- limit to grow a cache of dirty pages for reuse in the current transaction;
- limit of a pre-allocated memory items for dirty pages;
- limit of dirty pages for a write transaction;
- initial allocation size for dirty pages list of a write transaction;
- maximal part of the dirty pages may be spilled when necessary;
- minimal part of the dirty pages should be spilled when necessary;
- how much of the parent transaction dirty pages will be spilled while start each child transaction;
- Unlimited/Dynamic size of retired and dirty page lists (https://libmdbx.dqdkfa.ru/dead-github/issues/123).
- Added
-p
option (purge subDB before loading) tomdbx_load
tool. - Reworked spilling of large transaction and committing of nested transactions:
- page spilling code reworked to avoid the flaws and bugs inherited from LMDB;
- limit for number of dirty pages now is controllable at runtime;
- a spilled pages, including overflow/large pages, now can be reused and refunded/compactified in nested transactions;
- more effective refunding/compactification especially for the loosed page cache.
- Added
MDBX_ENABLE_REFUND
andMDBX_PNL_ASCENDING
internal/advanced build options. - Added
mdbx_default_pagesize()
function. - Better support architectures with a weak/relaxed memory consistency model (ARM, AARCH64, PPC, MIPS, RISC-V, etc) by means C11 atomics.
- Speed up page number lists and dirty page lists (https://libmdbx.dqdkfa.ru/dead-github/issues/132).
- Added
LIBMDBX_NO_EXPORTS_LEGACY_API
build option.
Fixes:
- Fixed missing cleanup (null assigned) in the C++ commit/abort (https://libmdbx.dqdkfa.ru/dead-github/pull/143).
- Fixed
mdbx_realloc()
for case of nullptr andMDBX_WITHOUT_MSVC_CRT=ON
for Windows. - Fixed the possibility to use invalid and renewed (closed & re-opened, dropped & re-created) DBI-handles (https://libmdbx.dqdkfa.ru/dead-github/issues/146).
- Fixed 4-byte aligned access to 64-bit integers, including access to the
bootid
meta-page's field (https://libmdbx.dqdkfa.ru/dead-github/issues/153). - Fixed minor/potential memory leak during page flushing and unspilling.
- Fixed handling states of cursors's and subDBs's for nested transactions.
- Fixed page leak in extra rare case the list of retired pages changed during update GC on transaction commit.
- Fixed assertions to avoid false-positive UB detection by CLANG/LLVM (https://libmdbx.dqdkfa.ru/dead-github/issues/153).
- Fixed
MDBX_TXN_FULL
and regressiveMDBX_KEYEXIST
during large transaction commit withMDBX_LIFORECLAIM
(https://libmdbx.dqdkfa.ru/dead-github/issues/123). - Fixed auto-recovery (
weak->steady
with the same boot-id) when Database size at last weak checkpoint is large than at last steady checkpoint. - Fixed operation on systems with unusual small/large page size, including PowerPC (https://libmdbx.dqdkfa.ru/dead-github/issues/157).
v0.9.2 at 2020-11-27
Acknowledgements:
- Jens Alfke (Mobile Architect at Couchbase) for NimDBX.
- Clément Renault (CTO at MeiliSearch) for mdbx-rs.
- Alex Sharov (Go-Lang Teach Lead at TurboGeth/Ethereum) for an extreme test cases and bug reporting.
- George Hazan (CTO at Miranda NG) for bug reporting.
- Positive Technologies for funding and The Standoff.
Added features:
- Provided package for buildroot.
- Binding for Nim is available now by Jens Alfke.
- Added
mdbx_env_delete()
for deletion an environment files in a proper and multiprocess-safe way. - Added
mdbx_txn_commit_ex()
with collecting latency information. - Fast completion pure nested transactions.
- Added
LIBMDBX_INLINE_API
macro and inline versions of some API functions. - Added
mdbx_cursor_copy()
function. - Extended tests for checking cursor tracking.
- Added
MDBX_SET_LOWERBOUND
operation formdbx_cursor_get()
.
Fixes:
- Fixed missing installation of
mdbx.h++
. - Fixed use of obsolete
__noreturn
. - Fixed use of
yield
instruction on ARM if unsupported. - Added pthread workaround for buggy toolchain/cmake/buildroot.
- Fixed use of
pthread_yield()
for non-GLIBC. - Fixed use of
RegGetValueA()
on Windows 2000/XP. - Fixed use of
GetTickCount64()
on Windows 2000/XP. - Fixed opening DB on a network shares (in the exclusive mode).
- Fixed copy&paste typos.
- Fixed minor false-positive GCC warning.
- Added workaround for broken
DEFINE_ENUM_FLAG_OPERATORS
from Windows SDK. - Fixed cursor state after multimap/dupsort repeated deletes (https://libmdbx.dqdkfa.ru/dead-github/issues/121).
- Added
SIGPIPE
suppression for internal thread duringmdbx_env_copy()
. - Fixed extra-rare
MDBX_KEY_EXIST
error duringmdbx_commit()
(https://libmdbx.dqdkfa.ru/dead-github/issues/131). - Fixed spilled pages checking (https://libmdbx.dqdkfa.ru/dead-github/issues/126).
- Fixed
mdbx_load
for 'plain text' and without-s name
cases (https://libmdbx.dqdkfa.ru/dead-github/issues/136). - Fixed save/restore/commit of cursors for nested transactions.
- Fixed cursors state in rare/special cases (move next beyond end-of-data, after deletion and so on).
- Added workaround for MSVC 19.28 (Visual Studio 16.8) (but may still hang during compilation).
- Fixed paranoidal Clang C++ UB for bitwise operations with flags defined by enums.
- Fixed large pages checking (for compatibility and to avoid false-positive errors from
mdbx_chk
). - Added workaround for Wine (https://github.com/miranda-ng/miranda-ng/issues/1209).
- Fixed
ERROR_NOT_SUPPORTED
while opening DB by UNC pathnames (https://github.com/miranda-ng/miranda-ng/issues/2627). - Added handling
EXCEPTION_POSSIBLE_DEADLOCK
condition for Windows.
v0.9.1 2020-09-30
Added features:
- Preliminary C++ API with support for C++17 polymorphic allocators.
- Online C++ API reference by Doxygen.
- Quick reference for Insert/Update/Delete operations.
- Explicit
MDBX_SYNC_DURABLE
to sync modes for API clarity. - Explicit
MDBX_ALLDUPS
andMDBX_UPSERT
for API clarity. - Support for read transactions preparation (
MDBX_TXN_RDONLY_PREPARE
flag). - Support for cursor preparation/(pre)allocation and reusing (
mdbx_cursor_create()
andmdbx_cursor_bind()
functions). - Support for checking database using specified meta-page (see
mdbx_chk -h
). - Support for turn to the specific meta-page after checking (see
mdbx_chk -h
). - Support for explicit reader threads (de)registration.
- The
mdbx_txn_break()
function to explicitly mark a transaction as broken. - Improved handling of corrupted databases by
mdbx_chk
utility andmdbx_walk_tree()
function. - Improved DB corruption detection by checking parent-page-txnid.
- Improved opening large DB (> 4Gb) from 32-bit code.
- Provided
pure-function
andconst-function
attributes to C API. - Support for user-settable context for transactions & cursors.
- Revised API and documentation related to Handle-Slow-Readers callback feature.
Deprecated functions and flags:
- For clarity and API simplification the
MDBX_MAPASYNC
flag is deprecated. Just useMDBX_SAFE_NOSYNC
orMDBX_UTTERLY_NOSYNC
instead of it. MDBX_oom_func
,mdbx_env_set_oomfunc()
andmdbx_env_get_oomfunc()
replaced withMDBX_hsr_func
,mdbx_env_get_hsr
andmdbx_env_get_hsr()
.
Fixes:
- Fix
mdbx_strerror()
forMDBX_BUSY
error (no error description is returned). - Fix update internal meta-geo information in read-only mode (
EACCESS
orEBADFD
error). - Fix
mdbx_page_get()
null-defer when DB corrupted (crash bySIGSEGV
). - Fix
mdbx_env_open()
for re-opening after non-fatal errors (mdbx_chk
unexpected failures). - Workaround for MSVC 19.27
static_assert()
bug. - Doxygen descriptions and refinement.
- Update Valgrind's suppressions.
- Workaround to avoid infinite loop of 'nested' testcase on MIPS under QEMU.
- Fix a lot of typos & spelling (Thanks to Josh Soref for PR).
- Fix
getopt()
messages for Windows (Thanks to Andrey Sporaw for reporting). - Fix MSVC compiler version requirements (Thanks to Andrey Sporaw for reporting).
- Workarounds for QEMU's bugs to run tests for cross-builded library under QEMU.
- Now C++ compiler optional for building by CMake.
v0.9.0 2020-07-31 (not a release, but API changes)
Added features:
- Online C API reference by Doxygen.
- Separated enums for environment, sub-databases, transactions, copying and data-update flags.
Deprecated functions and flags:
- Usage of custom comparators and the
mdbx_dbi_open_ex()
are deprecated, since such databases couldn't be checked by themdbx_chk
utility. Please use the value-to-key functions to provide keys that are compatible with the built-in libmdbx comparators.
2020-07-06
- Added support multi-opening the same DB in a process with SysV locking (BSD).
- Fixed warnings & minors for LCC compiler (E2K).
- Enabled to simultaneously open the same database from processes with and without the
MDBX_WRITEMAP
option. - Added key-to-value,
mdbx_get_keycmp()
andmdbx_get_datacmp()
functions (helpful to avoid using custom comparators). - Added
ENABLE_UBSAN
CMake option to enabling the UndefinedBehaviorSanitizer from GCC/CLANG. - Workaround for CLANG bug.
- Returning
MDBX_CORRUPTED
in case all meta-pages are weak and no other error. - Refined mode bits while auto-creating LCK-file.
- Avoids unnecessary database file re-mapping in case geometry changed by another process(es).
From the user's point of view, the
MDBX_UNABLE_EXTEND_MAPSIZE
error will now be returned less frequently and only when using the DB in the current process really requires it to be reopened. - Remapping on-the-fly and of the database file was implemented. Now remapping with a change of address is performed automatically if there are no dependent readers in the current process.
2020-06-12
- Minor change versioning. The last number in the version now means the number of commits since last release/tag.
- Provide ChangeLog file.
- Fix for using libmdbx as a C-only sub-project with CMake.
- Fix
mdbx_env_set_geometry()
for case it is called from an opened environment outside of a write transaction. - Add support for huge transactions and
MDBX_HUGE_TRANSACTIONS
build-option (defaultOFF
). - Refine LTO (link time optimization) for clang.
- Force enabling exceptions handling for MSVC (
/EHsc
option).
2020-06-05
- Support for Android/Bionic.
- Support for iOS.
- Auto-handling
MDBX_NOSUBDIR
while opening for any existing database. - Engage github-actions to make release-assets.
- Clarify API description.
- Extended keygen-cases in stochastic test.
- Fix fetching of first/lower key from LEAF2-page during page merge.
- Fix missing comma in array of error messages.
- Fix div-by-zero while copy-with-compaction for non-resizable environments.
- Fixes & enhancements for custom-comparators.
- Fix
MDBX_WITHOUT_MSVC_CRT
option and missingntdll.def
. - Fix
mdbx_env_close()
to work correctly called concurrently from several threads. - Fix null-deref in an ASAN-enabled builds while opening the environment with error and/or read-only.
- Fix AddressSanitizer errors after closing the environment.
- Fix/workaround to avoid GCC 10.x pedantic warnings.
- Fix using
ENODATA
for FreeBSD. - Avoid invalidation of DBI-handle(s) when it just closes.
- Avoid using
pwritev()
for single-writes (up to 10% speedup for some kernels & scenarios). - Avoiding
MDBX_UTTERLY_NOSYNC
as result of flags merge. - Add
mdbx_dbi_dupsort_depthmask()
function. - Add
MDBX_CP_FORCE_RESIZEABLE
option. - Add deprecated
MDBX_MAP_RESIZED
for compatibility. - Add
MDBX_BUILD_TOOLS
option (defaultON
). - Refine
mdbx_dbi_open_ex()
to safe concurrently opening the same handle from different threads. - Truncate clk-file during environment closing. So a zero-length lck-file indicates that the environment was closed properly.
- Refine
mdbx_update_gc()
for huge transactions with small sizes of database page. - Extends dump/load to support all MDBX attributes.
- Avoid upsertion the same key-value data, fix related assertions.
- Rework min/max length checking for keys & values.
- Checking the order of keys on all pages during checking.
- Support
CFLAGS_EXTRA
make-option for convenience. - Preserve the last txnid while copying with compactification.
- Auto-reset running transaction in mdbx_txn_renew().
- Automatically abort errored transaction in mdbx_txn_commit().
- Auto-choose page size for large databases.
- Rearrange source files, rework build, options-support by CMake.
- Crutch for WSL1 (Windows subsystem for Linux).
- Refine install/uninstall targets.
- Support for Valgrind 3.14 and later.
- Add check-analyzer check-ubsan check-asan check-leak targets to Makefile.
- Minor fix/workaround to avoid UBSAN traps for
memcpy(ptr, NULL, 0)
. - Avoid some GCC-analyzer false-positive warnings.
2020-03-18
- Workarounds for Wine (Windows compatibility layer for Linux).
MDBX_MAP_RESIZED
renamed toMDBX_UNABLE_EXTEND_MAPSIZE
.- Clarify API description, fix typos.
- Speedup runtime checks in debug/checked builds.
- Added checking for read/write transactions overlapping for the same thread, added
MDBX_TXN_OVERLAPPING
error andMDBX_DBG_LEGACY_OVERLAP
option. - Added
mdbx_key_from_jsonInteger()
,mdbx_key_from_double()
,mdbx_key_from_float()
,mdbx_key_from_int64()
andmdbx_key_from_int32()
functions. Seemdbx.h
for description. - Fix compatibility (use zero for invalid DBI).
- Refine/clarify error messages.
- Avoids extra error messages "bad txn" from mdbx_chk when DB is corrupted.
2020-01-21
- Fix
mdbx_load
utility for custom comparators. - Fix checks related to
MDBX_APPEND
flag insidemdbx_cursor_put()
. - Refine/fix dbi_bind() internals.
- Refine/fix handling
STATUS_CONFLICTING_ADDRESSES
. - Rework
MDBX_DBG_DUMP
option to avoid disk I/O performance degradation. - Add built-in help to test tool.
- Fix
mdbx_env_set_geometry()
for large page size. - Fix env_set_geometry() for large pagesize.
- Clarify API description & comments, fix typos.
2019-12-31
- Fix returning MDBX_RESULT_TRUE from page_alloc().
- Fix false-positive ASAN issue.
- Fix assertion for
MDBX_NOTLS
option. - Rework
MADV_DONTNEED
threshold. - Fix
mdbx_chk
utility for don't checking some numbers if walking on the B-tree was disabled. - Use page's mp_txnid for basic integrity checking.
- Add
MDBX_FORCE_ASSERTIONS
built-time option. - Rework
MDBX_DBG_DUMP
to avoid performance degradation. - Rename
MDBX_NOSYNC
toMDBX_SAFE_NOSYNC
for clarity. - Interpret
ERROR_ACCESS_DENIED
fromOpenProcess()
as 'process exists'. - Avoid using
FILE_FLAG_NO_BUFFERING
for compatibility with small database pages. - Added install section for CMake.
2019-12-02
- Support for Mac OSX, FreeBSD, NetBSD, OpenBSD, DragonFly BSD, OpenSolaris, OpenIndiana (AIX and HP-UX pending).
- Use bootid for decisions of rollback.
- Counting retired pages and extended transaction info.
- Add
MDBX_ACCEDE
flag for database opening. - Using OFD-locks and tracking for in-process multi-opening.
- Hot backup into pipe.
- Support for cmake & amalgamated sources.
- Fastest internal sort implementation.
- New internal dirty-list implementation with lazy sorting.
- Support for lazy-sync-to-disk with polling.
- Extended key length.
- Last update transaction number for each sub-database.
- Automatic read ahead enabling/disabling.
- More auto-compactification.
- Using -fsanitize=undefined and -Wpedantic options.
- Rework page merging.
- Nested transactions.
- API description.
- Checking for non-local filesystems to avoid DB corruption.
For early changes see the git commit history.