Dual locking: instead of using one global lock for a joedb file, this
version locks head and tail separately. This allows a much nicer handling
of concurrent access to files:
Journal construction locks the head only, so it does not block if a
transaction is in progress or the file was opened in exclusive mode since
both of those situations lock the tail only.
Concurrent reads use a shared lock on the head of the file to read the
checkpoint, and can be blocked only during very short periods of time
when a writable journal is constructed or when the checkpoint is
modified.
Write access to exclusive and shared files is completely unified, so:
Connection and File_Connection can now handle both exclusive
and shared files.
Local_Connection is removed since it can be replaced by a plain
Connection.
New CURL_File allows opening any URL as a read-only file.
Incompatibilities with previous version:
In Posix environments, locking changed from using flock to using
fcntl. Those two locking mechanisms are not compatible in Linux, so
it is important to avoid mixing joedb versions because they may not
understand each-other’s locks. Windows and MacOS do not have this
problem.
The network protocol changed, and is not compatible with the previous
version.
2024-04-23: 8.0.1
Fix missing test for fsync error.
2024-04-21: 8.0.0
joedb now uses C++17, and is not compatible with C++11 any more.
joedb_server now takes a client as parameter, instead of a file. This
gives much more flexibility, and allows:
chaining servers for synchronous remote backup;
more generally, creating a tree of multiple database replicas;
local programs running on the same machine as the server can access the
database file directly, without having to use a network connection to the
server.
Some changes to better handle very large databases:
The server does not buffer a whole push before writing it to disk any
more. This saves memory and time in case of a large push. If a connection
fails in the middle of a push, the written data is not erased. But it is
not checkpointed, and may be overwritten by a subsequent push.
Better timeouts: the timeout is not for the whole transaction, but for
any network activity during the transaction. So, a very long transaction
because of a very large push or pull will not time out if there is
continuous network activity. Also, previous versions did not check for
timeouts in the middle of a push. This is now fixed.
Performance optimizations:
File_Connection is now about 10 times faster.
Large string reads are also much faster.
Improved networking performance by fusing small consecutive socket
writes. Previous version could be hit hard by Nagle’s algorithm and
delayed ACKs. This version uses ip::tcp::no_delay.
joedbc produces a new Pullable_Database, similar to
Readonly_Database, but the file is not closed at the end of the
constructor, and it is possible to pull new data in case of a concurrent
update.
Classes that write a journal (joedb::Writable_Journal,
Generic_File_Database, File_Database, Client) now have two
extra parameters:
check indicates the behaviour in case the file contains an incomplete
transaction. It should be equal to either:
joedb::Readable_Journal::check::all (the default) fails if the file
contains data after the checkpoint.
commit_level indicates the default commit level for checkpoints.
Default is joedb::Commit_Level::no_commit. See Checkpoints for details.
id_of_x is now a literal type. All its member functions are
constexpr.
Minor fixes and improvements.
2023-08-15: 7.1.1
Bug fix of previous version: large pulls (>256kb) from a
Readonly_File_Connection to an SFTP_File could fail.
2023-07-07: 7.1
New SFTP_File, and Readonly_File_Connection. Combining these
classes allows read-only connection to a remote file without running a
joedb server on the remote machine. This is convenient for periodic
backups. SFTP does not support file locking in practice, so writing via
SFTP is not implemented.
New joedb_push tool, with new interesting features
such as the ability to follow a file or push to a remote backup server.
Removed some useless ssh code that was left over from the old serverless
sftp connection. ssh::Thread_Safe_Sesion is renamed to
ssh::Session, and its constructor is faster than before because there
is no sftp any more.
No more implicit conversion from compiled row id to integer or boolean.
With this new version, explicit methods must be used: is_null() or
is_not_null() to test if a reference is null or not, and get_id()
to convert to an integer. This ensures stronger typing, and prevents
bug-prone implicit conversions.
Type-safe vector update of references. A range of id_of_x is passed
instead of the generic Record_Id.
No more endianness conversions: on big-endian machines joedb reads and
writes data in big-endian format.
Minor fixes and improvements
2021-09-15: 6.0
new Local_Connection class for local serverless concurrent access to a file.
SHA-256 is used to compare the client database with the server database at
connection time. This prevents pulling into the wrong file. This also
allows making offline modifications to a local database, and pushing them
later to a remote server.
Re-organize file classes:
File_Slice is removed. All file classes can be sliced now.
Stream_File uses a std::streambuf instead of std::iostream.
new Interpreted_File can read joedbi commands directly.
Exception-safe transactions: if any exception is
thrown by a client while writing, then none of what was written since the
previous lock-pull will be pushed to the server. The previous approach,
based on a Lock object, was defective and is not available any more.
New approach to vector updates that allows testing for
write errors (previous version wrote data in a destructor, which does not
allow testing for errors).
Databases must be explictly checkpointed before destruction. The destructor
won’t checkpoint any more, because this would risk checkpointing a failed
buffer flush, and because write errors cannot be handled properly in
destructors.
Defining the JOEDB_PORTABLE macro builds joedb with portable C++ only
(no file locking, no fsync). With this option, joedb can be used on the
PlayStation 4 and the Nintendo Switch.
Unlike in Linux, fseek and ftell are 32-bit in Windows. So the
previous version could not handle files larger than 2^31 bytes. This is
now fixed, and very large files can be used in Windows.
Unlike Linux, Windows does no print any information when a program is
terminated by an exception. Joedb tools in this version catch all
exceptions, and print them before quitting.
Main version number incremented because of one minor change: custom
functions are now member of Generic_File_Database instead of the
File_Database class.
dense table storage is more memory-efficient in the interpreter
Minor fixes and improvements
2017-01-18: 2.0
Exceptions everywhere: no more error codes, no more bad states, better
diagnostics.
Safety: several safety checks were added. This version was thoroughly
fuzzed, and should not crash on any input file. Many assertions were added
to detect data-manipulation errors (double delete, double insert, reading
invalid rows, etc.).
Better handling of read-only files and locking. A file opened for writing
can now be opened for reading by other processes. Readers won’t be updated
by changes made by the writer, but it is still more convenient than before.
The compiler can produce a rudimentary C wrapper around the C++ classes.