LENTS (Local Extendable Nested Tagging System) is an implementation of ENTS written in C that uses a custom binary file format as a database. It is the most hackerish version of ENTS, has the smallest database size, and was made as a low-level learning project. It is also named for the liturgical season it was started in, Lent.
LENTS usage
process
The output for lents process tags.yaml will always be to
a .lents hidden dot-file. This output cannot be changed,
for the sake of parsing.
The LENTS file format
- Terminology and stylistic choices
- Some set theory will be used in combination with colloquial English because it is the best way to communicate some things clearly and unambiguously
- Hex-line: refers to a standard 16-byte row in a hex dump
- Metadata: is a common term meaning “data about the data” used here.
- Ipsodata: contrasted with metadata. “Data that is
the data itself”. Commonly used English terms for “the data itself” such
as “core data”, “primary data”, “source data”, and “raw data” do not
capture what I want to mean here, the data itself which is
opposed to the data-about-the-data. Uses the Latin root
“ipso”, meaning “itself” as a prefix.
- Offset: When offset is said here, I am referring to only the length of the offset and not the content of the offset.
- Over-arching design choices for LENTS
- LENTS is a directed acyclic graph structure where hierarchically organized tags (nodes with parent-child relationships) map to files (destination nodes)
- The file format is designed for human-readability in hex dump
- File Header (32 Bytes, or 2 hex-lines)
- LENTS magic number in unicode
- Version number
- Offsets
- Start of tags ipsodata
- Start of file-to-tags metadata
- Start of file-to-tags ipsodata
- Data
Current implementations of LENTS use a structured a two-level index to two offset lookup tables (tags & file->tags) and the first table (tags) also has tag-to-files relationships. This is the most space-efficient representation for sparse relationships, but as relationships get denser there is a threshold after which it is more space-efficient to use a table encoding. The table encoding will be included in a future version of LENTS.
- Tags Data
Each tag ipsodata can vary significantly in length, so an offset table is used to save lots of file space at minimal expense of lookup time. All the attributes in the ipsodata can also vary in length from eachother, so a “tag individual ipsodata part length” field is needed in the metadata. Tag ipsodata offset is the distance measured from (3.3.a). The tag UUID is not neccassary for the tag ipsodata, but is included for debug purposes.
- Tags Metadata =
{
Tag Metadata}
- Tag Metadata = {Tag UUID, Tag ipsodata offset}
| Name | Size | Description |
| Tag Hash UID | 7 bytes | Hash of the tag name |
| Space for flags | 1 byte | Unused for now. May be used in future versions of LENTS while still being backwards compatible |
| Tag ipsodata offset | 4 bytes | Value of how far away from start-of-tags-ipsodata (3.3.a) the tag ipsodata is |
- Tags Ipsodata =
{
Tag Ipsodata}
- Tag Ipsodata = {Tag UUID, Tag name, Tag Ancestry UUIDs, Tag Children UUIDs}
| Name | Size | Description |
| Tag Hash UID | 7 bytes | Hash of the tag name |
| Children Count | 1 byte | Counts the number of children this tag has. Used to calculate offset. This gives a max value of 255, but please be more organized and use nesting instead. The only reason it goes up to 255 is for the case of having each country as a tag. |
| Tag Immediate Parent UID | 7 bytes | For traversal upwards |
| Name Length | 4 bits | Values 0-15. Each bit is multiplied by 4 and then has 4 added to it (for the case of 0 - you cannot have a nameless tag) to get a final value between 4 and 64. Half of the byte with Tag Type Flags |
| Tag Type Flags | 4 bits | Future feature to be implemented. Half of the byte with Name Length |
| Tag-to-Files relationship lenghts | 2 bytes | Currently has value from 0-2^16. There is space to be expanded for larger systems |
| Blank Space | 6 bytes | For storing values in future versions. Also for the sake of visual padding when viewed from a hex dump |
- File-to-tags Relationship Data
The same heuristics about length variation and UUID-inclusion from (4) apply here. File-to-tags metadata starts at (3.3.b) and file ipsodata offset is the distance measured from (3.3.c).
- Files Metadata =
{
File Metadata}
- File Metadata = {File UUID, File individual ipsodata part length, File ipsodata offset}
- Files Ipsodata =
{
File Ipsodata}
- File Ipsodata = {File UUID, Tags UUIDs}