 |
BELFEM 0.9.0
Berkeley Lab Finite Element Framework
|
|
BELFEM 0.9.0
Berkeley Lab Finite Element Framework
|
Date: 2026-08-08 Purpose: Standards for nomenclature, memory management, container selection, and design patterns to ensure high-performance execution and developer clarity
Revision History:
BELFEM (Berkeley Lab Finite Element Framework) is designed for high-performance finite element simulations on HPC systems. Every design decision prioritizes:
This document explains the "why" behind BELFEM's coding conventions. For contributors joining from modern C++ projects emphasizing safety and smart pointers, understanding these principles is essential.
Target audience: Developers contributing to BELFEM, code reviewers, and anyone wondering why the code looks different from typical C++14/17 practices.
Provenance of claims: Facts in this document are anchored to the source with file:line citations, current as of 2026-08-08. Code blocks are either verbatim excerpts (cited) or explicitly labeled pseudocode. Where a claim could not be verified from the repository, it is either cited to external literature and labeled a literature heuristic, or it has been removed — see the Source Reconciliation appendix.
Philosophy: In HPC, a 5% performance regression across a large simulation costs real money and time. BELFEM optimizes for the release build, then adds safety checks in debug builds.
In practice:
Exceptions status (honest statement): BELFEM's design intent is to be compatible with exception-free production builds — no throw on hot paths, error macros that abort in release. However, no build configuration currently sets -fno-exceptions: the flag appears in no CMake file, and src/core/assert.hpp:191 contains a throw that is compiled into every default build, so an actual -fno-exceptions build would not compile today. Treat "exception-free production" as a design goal, not a property of the current build system.
Testing posture (honest statement): Tests are GoogleTest suites run via make check, gated on USE_TEST which defaults ON (option( USE_TEST "Build Tests" ON ) in CMakeLists.txt; release policy since 2026-08-14). Since 2026-08-30 a GitLab CI server runs the full suite nightly, which retires the older statement that nothing ran the tests automatically.
Three qualifications keep this an honest statement rather than a claim of safety. First, the pipeline definition (.gitlab-ci.yml at the repository root) runs only on a scheduled or manually started pipeline, on a single runner tagged belfem-local, so the tree records what the nightly builds and tests but not the hardware it runs on, and a clone cannot reproduce it. Second, there is still no sanitizer configuration: -fsanitize appears in no CMake file, and Valgrind is invoked only by hand. Third, and most important, a passing nightly measures the suite, not the code. This project's own incident catalog records 539 incidents in its locked catalog (addenda run to INC-564) of which 280 were code defects and 38 were closed by adding a test; the suite is not yet the thing that finds our bugs, and a green pipeline must not be read as though it were.
The check-removal bargain — "we may compile the bounds checks out because testing catches the bugs" — is therefore partly earned. The automation half now exists. The coverage half does not: the modules carrying the heaviest recorded defect load are also the thinnest covered (src/mesh is 52,318 source lines against 915 test lines). Full payment still waits on ASan/UBSan jobs running on every change, and on coverage reaching the subsystems that actually break; wiring both up is a tracked proposal (see appendix).
Contrast with modern C++: Modern C++ emphasizes "safety by default" (smart pointers, exceptions, bounds-checked containers). BELFEM inverts this: speed by default, safety in development.
Philosophy: Every abstraction layer adds overhead. BELFEM stays close to "bare metal" when it matters.
In practice:
Why: HPC centers use diverse architectures (x86, ARM, Power). Minimal dependencies ensure portability and give full control over memory layout.
Philosophy: Hidden behavior (implicit conversions, reference counting, magic destructors) makes performance analysis impossible.
In practice:
Philosophy: FEM code implements published algorithms. The code should resemble the equations in papers.
In practice:
Example: A stiffness matrix assembly loop should look like the formula in Bathe, not like Java enterprise code.
BELFEM uses a strict prefix system to eliminate ambiguity and improve code readability at a glance.
| Prefix | Meaning | Scope | Example |
|---|---|---|---|
| a | Argument | Function parameters (all public APIs) | aTarget, aNumRows, aMessage |
| t | Temporary | Local variables (function scope) | tCount, tStatus, tBuffer |
| m | Member | Class data members (private/protected) | mMatrix, mInfoLevel, mCommRank |
| g | Global | Framework-wide globals (extern singletons) | gLog, gComm, gTbulk |
1. Readability at a glance (pseudocode)
You know instantly: aTarget is a parameter, tCommTag is local.
2. Prevents shadowing bugs (pseudocode)
3. Consistency across 300k lines All BELFEM modules (core, comm, fem, mesh, etc.) use identical conventions. Switching between modules requires zero mental context switch.
When implementing numerical algorithms directly from literature, readability trumps strict prefixes.
Allowed in mathematical kernels:
In src/math and src/physics kernels the a/t prefixes may be dropped entirely so the code reads like the derivation (see CLAUDE.md for the canonical thermophysical symbol table). The m and g prefixes always still apply.
Where the kernel/framework boundary lies: prefixes resume the moment a variable is no longer part of the mathematical expression. The same function can contain both regimes:
(pseudocode)
Types: CamelCase (e.g., DynamicBitset, Matrix, InfoLevel)
Functions: lowercase_with_underscores (e.g., set_size(), comm_tag(), string_to_bool()). Exception: capitalized mathematical symbols such as T (temperature) are preserved in the physics modules to avoid colliding with t (time).
Constants: BELFEM_UPPERCASE for macros (e.g., BELFEM_EPSILON, BELFEM_INT64)
Files: the stem follows the prefix, matching what the file defines:
Documentation files are always lowercase_with_underscores.md.
BELFEM provides custom container wrappers around STL for consistency, safety, and FEM-specific features.
The workhorse — use for almost all dynamic arrays.
Cell<T> is a thin wrapper around std::vector<T> (src/containers/cl_Cell.hpp:44). The argument for it is interface and discipline, not allocation performance — allocation is std::vector's in both cases. What the wrapper buys:
Example (real API — note size() and the free-function unique):
Use ONLY for linear algebra — not general-purpose arrays.
Why separate from Cell?
Example (pseudocode):
Don't use Vector<T> for ID lists, node collections, etc. — use Cell<T> instead.
BELFEM matrices are column-major (Fortran convention). Column j is contiguous in memory; row i is strided.
Why column-major:
Backend caveat — data() interpretation:
Under Armadillo, the storage is exactly A(i, j) == data[j * nrows + i], and capacity() equals n_rows * n_cols (cl_AR_Matrix.hpp:224-226).
Under Blaze, columns are padded for SIMD alignment (padding is on by default and BELFEM does not disable it), so the inter-column stride — spacing(), the "leading dimension" in BLAS terms — may exceed nrows, and capacity() exceeds n_rows * n_cols. The LAPACK wrappers already honor this: "passing n_rows() instead would silently corrupt the result" (src/linalg/lapack/lapacktools.hpp:150-153).
Rule: always use the A(i, j) accessor for element access. Never compute offsets from data() directly. Reserve data() for bulk operations (BLAS/LAPACK calls that take a leading-dimension argument, MPI transfers that treat the buffer opaquely).
For MPI, the comm layer transmits the padded footprint of the current shape — spacing() * n_cols elements from data() — so the padded layout round-trips bit-identically between ranks running the same binary (src/comm/commtools.hpp documents this design in place; Blaze's spacing is a pure function of the row count, so both sides always agree). It deliberately does not transmit capacity(): a matrix that shrank keeps its old, larger allocation, and that stale count would overflow the exact-fit buffer on the receiving side (defect found and fixed 2026-08-09; the receive paths now also guard capacity() >= transfer length with BELFEM_ERROR). A Blaze build's serialized matrix is still NOT byte-compatible with an Armadillo build's — relevant only if matrices are checkpointed across builds with different backends.
Iteration order matters. The inner loop must vary the row index for cache-friendly access:
For matrices larger than the cache, the slow form misses on every inner-loop access — with 8-byte reals on 64-byte cache lines, up to 8 loads per line instead of 1. (Literature heuristic, not a BELFEM benchmark: Drepper, "What Every Programmer Should Know About Memory", §3.3; Agner Fog, "Optimizing software in C++", §9.)
Watch out: many C++ libraries default to row-major (Eigen, NumPy, the default mental model from C-array pedagogy). BELFEM does not.
DynamicBitset — Runtime-sized bit array
ShiftRegister<T> — Fixed-depth history buffer
Bitset<N> — Compile-time sized bit array
StringList — C-compatible string array
Reason 1: One interface, one discipline. Every module uses the same containers with the same conventions (set_size, push, operator(), data()), so switching modules costs nothing. The wrapper is also the single place where debug checks, FEM utilities, and future instrumentation live. This is the "deep module" argument (Ousterhout): a small interface hiding real functionality beats re-deciding vector idioms at every call site.
Reason 2: Debug safety with release speed. operator() asserts in debug and compiles to raw access in release — std::vector::operator[] gives you only the latter, at() only the former (always-on, throwing).
Reason 3: Backend independence. Vector/Matrix code is written once against the BELFEM API and runs on either Armadillo or Blaze; the wrapper absorbs the differences (padding, memptr() vs data(), resize semantics).
Reason 4: Errors defined out of existence. data() on an empty container is well-defined and composes with zero-count MPI calls; sized constructors initialize with sentinels; utilities like unique() and find_index_in_unique_cell() encode the FEM idiom once, correctly.
Every graph vertex — and therefore every mesh entity (Node, Edge, Face, Element, Facet) — carries an 8-slot flag bitset (graph::Vertex::mFlags, uint8_t; flag( aIndex = 0 ) / unflag( aIndex = 0 ) / is_flagged( aIndex = 0 )).
Prefer flags over Map lookups for marking, dedup, and visited-set logic on mesh entities: zero allocation, O(1) by construction, and the entity itself carries the state.
Slot convention:
Hygiene: flags are global mesh state. A function-local flag use must clear the touched range before the gather (stale flags from an earlier user must not leak in) and after it (the next user must find a clean slate). The pre-clear is required even on numbered slots — no convention guarantees a slot was left clean.
The lead principle is not "manual is faster than smart pointers." It is: allocation, ownership, and layout decisions are made deliberately during setup, so that the solve phase runs on stable, compact, predictable data structures. Manual allocation is one tool for that; so are the backend allocators, the STL wrapped inside Cell, and — in setup-scope code — smart pointers.
Two rules make the deliberateness concrete:
The entity rule. Objects instantiated at mesh scale (nodes, elements, facets, edges, faces) contain only fixed-size state, compact handles/indices, and non-owning pointers. Variable-size storage belongs to aggregate owners or flattened tables — never to the entity itself growing ad hoc. (An element's node-pointer array is fixed at its type's node count, allocated once in the constructor: cl_ElementTemplate.hpp:383,404,426.)
The ownership rule. Ownership is hierarchical and established at setup: the mesh owns its entities, blocks own their elements, the kernel owns its managers. Shared ownership is banned in computational data structures. A raw T* is the explicit non-owning borrow.
What manual/deliberate allocation buys, stated precisely:
What it does NOT buy (claims we no longer make):
Ownership is expressed by convention, in the spirit of the C++ Core Guidelines' owner<T*> / raw-T* split:
The sanctioned owning-holder pattern, verbatim from the tree — SideSet holds both flavors side by side (cl_SideSet.hpp:43-44, cl_SideSet.cpp:34-41):
Zero-cost owner<T*>/observer<T*> aliases plus a clang-tidy ownership check would turn this convention into tooling; that is a tracked proposal (appendix), not yet in the tree.
This is how mesh entities actually live and die — per-object new at setup, hierarchical delete at teardown. (There is no pool or arena allocator for mesh entities in BELFEM; an earlier revision of this document showed one as if it existed.)
1. Allocate & construct (setup). The element factory returns one heap object per element (cl_Element_Factory.cpp:65-193 — a pure creator with no state, cl_Element_Factory.hpp:32-41); the Gmsh reader news each node (cl_Mesh_GmshReader.cpp:277):
2. Use (solve). Entities are reached through non-owning Cell<T*> views and plain T* borrows. Nothing allocates or frees on this path.
3. Destroy (teardown). Ownership is layered: Mesh deletes nodes, edges, faces, vertices, and the containers of aggregates (cl_Mesh.cpp:205-290); Block deletes its elements (cl_Block.cpp:34-40); SideSet deletes its facets (cl_SideSet.cpp:34-41); a Facet deletes its wrapped element (cl_Facet.cpp:29-32). Each delete appears exactly once, in the destructor of the designated owner — ~Mesh() explicitly does not delete elements and facets, with comments marking whose job it is (cl_Mesh.cpp:256-266).
The pool pattern, done right. The one placement-new pool in the tree is ShiftRegister<T>: std::malloc for the buffer (cl_ShiftRegister.hpp:493), placement-new per slot (:109), explicit ->~T() per slot (:126), single std::free (:250-255). The rule it demonstrates: objects constructed by placement new are destroyed by an explicit destructor call plus free on the pool — calling delete on a pool member is undefined behavior. If a pooled/SoA element store is ever pursued (a legitimate post-release discussion), this is the discipline it must follow.
Rule of Five. Any class owning raw memory declares or deletes all five special members. The tree's raw owners are compliant and serve as the reference examples:
The real census (2026-08-08) is small and specific — and it is not "I/O and unit tests" (there are zero smart pointers in tests/ and zero in src/io/):
Rule of thumb (unchanged in spirit, corrected in letter): if the code runs inside the assembly/solve/communication loops, use the deliberate-allocation patterns above. If it runs once at setup or teardown, a smart pointer as a lifetime handle is acceptable — and for shared top-level objects, already established practice.
Problem: MPI counts are int-typed in the classic API; large mesh payloads exceed what a single message should carry.
Solution: the comm layer transparently splits every large transfer into chunks of gMaxCommChunkLength = 64 * 1024 elements of T — not bytes (src/comm/commtools.hpp:30). comm_split() computes the chunk list from the element count (commtools.cpp:80-92), and each chunk goes out as an MPI_Isend of count elements typed via comm_type<T>() (commtools.hpp:330-333), followed by a single MPI_Waitall.
Element types are primitives (or std::complex): comm_type<T>() is specialized only for arithmetic types and errors on anything else (src/comm/commtypes.hpp:38-204). There is no generic object serialization — a Cell of arbitrary structs cannot be sent without adding a specialization. Strings are flattened to Vector<char> (commtools.cpp:147-225).
Why chunking, when MPI-4 has large counts? MPI 4.0's MPI_Send_c family accepts MPI_Count payloads natively. Chunking is retained because production clusters still run older MPI stacks, and the chunk loop costs nothing measurable next to the transfer itself. When MPI-4 is the deployment floor, the chunk layer can collapse to a passthrough without touching any caller.
Distributing data from root to all ranks: payload size dictates the API.
| Payload | Function |
|---|---|
| Single scalar | broadcast(T&) |
| Small fixed-size payload (~≤ 8 entries, e.g., a properties tuple, a 3×3 matrix) | broadcast(...) |
| Large or variable-size Vector<T> or Cell<T> (mesh quantities, ephemeris columns) | share / receive (commtools.hpp:1548, :1648) |
| Matrix<T> of any size | broadcast(...) — no share( Matrix ) overload exists; for very large payloads use a manual send / receive loop |
broadcast is collective: every rank calls it with the same arguments, and the implementation packs the payload into a single unchunked MPI_Ibcast. Fine for small bounded payloads; observed to fail on large ones. For large variable-size Vector<T> or Cell<T> data, use the asymmetric share/receive pair, which chunks:
The if/else rank guard is required: unlike broadcast, share and receive are NOT collective. Calling either on the wrong rank is a deadlock. There is no share overload for scalars — scalar broadcast(T&) is always the right choice for single-element messages.
Matrix transfers ship the raw buffer. send(Matrix<T>&) transmits n_rows, n_cols, and the transfer length spacing() * n_cols, then that many elements of the raw buffer — deliberately including any backend padding, so the padded layout round-trips without disassembly (both ranks run the same binary, so the padding is identical for a given shape). Never capacity(): it can be stale-large after a shrink. The receive paths assert the transfer length fits the local buffer.
An earlier revision of this document showed BELFEM calling aligned_alloc(64, ...) for its vectors. That is not how it works. src/ contains no call to aligned_alloc or posix_memalign at all; alignment comes entirely from the linear-algebra backends:
So the honest statement is: numerical data is SIMD-aligned to whatever width the backend's build targets; nothing in BELFEM guarantees 64 bytes. BLAS kernels handle the residual peel/tail cases either way; the alignment mostly determines whether the vectorized core runs aligned loads. (The cache-line-split penalty for misaligned streaming access is a literature heuristic — Drepper §6.2, Agner Fog's optimization manuals — not a BELFEM measurement.)
If you ever hand-allocate aligned memory (currently no site does), the rules are:
Problem: std::vector<std::vector<T>> is not contiguous — each row is a separate allocation, so it cannot be sent as one buffer.
Solution: flatten to a single contiguous object.
This is also the data-oriented-design argument (Acton): the machine rewards arrays of plain data laid out for the access pattern, not graphs of small heap objects.
Principle: wrappers should compile to the same machine code as the raw implementation.
Example: Cell<T>'s operator() is an inlined std::vector access guarded by BELFEM_ASSERT (cl_Cell.hpp:148-168). Under USE_DEBUG=ON (-Og -g, DEBUG defined) the bounds check runs; under USE_DEBUG=OFF (NDEBUG) the assert expands to nothing and the accessor is a raw indexed load.
Verification: compare assembly with objdump -d or godbolt — the release accessor is indistinguishable from a raw array access.
No temporary Vector/Matrix in frequently-called member functions. Any method expected to run repeatedly at runtime allocates its scratch as members, sized once in the constructor (the caller may loop over it millions of times). Expression-template assignments into an already-sized member evaluate in place and are fine. Temporaries remain acceptable in constructors and one-off setup paths.
BELFEM uses a two-tier macro system plus a third, macro-free category for algorithmic failures.
The check's runtime cost is what justifies compiling it out — so where a requirement is validated follows how often the code runs:
A runtime requirement too expensive to check per-call is validated once at construction with BELFEM_ERROR — never demoted to debug-only just because the accessor is hot. The same precondition can legitimately be BELFEM_ERROR in a factory and BELFEM_ASSERT in the accessor it protects.
Solver non-convergence, line-search failure, a rejected timestep: these are not errors in either macro's sense. They are anticipated outcomes of numerical algorithms, and they return a status or trigger the retry path — timestep reduction, relaxation, controller fallback — exactly as the nonlinear controller already practices. Reaching for BELFEM_ERROR("did not converge") inside an algorithm that has a retry policy above it is a design error: it converts a recoverable state into a run abort.
Both funnel through belfem::assert::belfem_assert(...), which calls assert::error. That function prints the error box and then consults throw_on_error(): true throws the std::runtime_error, false calls error_abort(), which delegates to the comm module's comm_abort wrapper (declaration and full contract: src/comm/cl_Communicator.hpp:235-258; definition: src/comm/commtools.cpp:67). That wrapper aborts the job with MPI_Abort(MPI_COMM_WORLD, 1) — MPI_COMM_WORLD rather than gComm.world(), guarded by MPI_Initialized/MPI_Finalized — and otherwise falls through to std::abort(). BELFEM_ASSERT additionally compiles to nothing when assertions are inactive; BELFEM_ERROR is always expanded.
The flag is initialized to the build's compile-time behavior — throwing where assertions are active, aborting otherwise — so a debug run still throws and a production run still aborts, at any rank count, with no caller involvement. A debug run throws in parallel too, deliberately: that is what keeps a failed check inspectable while the other ranks are still alive. MPI_Abort is the production reaction. It is a test hook, not a configuration knob: a production run must keep the abort, because a throw that escapes main terminates one rank and leaves its peers blocked in a collective, and nothing in src/ catches BELFEM errors. The state deliberately lives in assert.cpp rather than as a header static, because assert::error is a function template instantiated in the calling translation unit — a per-TU copy would mean a test binary could not change the reaction of an already-compiled library.
The reason the hook exists: BELFEM_ERROR covers the failures that survive into release (file I/O, MPI, allocation, unsupported configuration), so those paths are worth unit-testing in the shipped configuration. Each test main calls set_throw_on_error( true ) after gComm.init, which makes EXPECT_THROW work under NDEBUG instead of aborting the whole test binary. Two consequences for test authors. First, guard assertion-tier tests with #if BELFEM_ASSERTIONS_ACTIVE, the macro exported by assert.hpp — not a restated #ifndef NDEBUG, which is not equivalent when both NDEBUG and DEBUG are defined. Second, the two predicates are distinct: a release binary in throw mode can catch a BELFEM_ERROR while BELFEM_ASSERT still expands to nothing. What such a test verifies is that the check fires with the right message, not that the process aborts; release and test runs take different reactions by construction.
Note precisely what drives the default abort-vs-throw split: the NDEBUG macro, not the exception model. Exceptions are currently enabled in every build configuration (see Core Principles); an uncaught std::bad_alloc from a failed std::vector reallocation in a debug build unwinds and terminates with a diagnosable message rather than aborting. Under the aspirational -fno-exceptions regime, allocation failure would become an immediate abort inside the library — one more reason that switch is a deliberate future decision, not a checkbox.
Design decision: BELFEM is deliberately not thread-safe internally.
Rationale:
What this means:
What IS thread-safe:
Design decision: BELFEM deliberately supports a single parallel programming model — pure MPI, one rank per core, application code single-threaded. Hybrid MPI+X is a cost we do not pay until a measured problem justifies it.
This is a scope decision, not a claim that hybrid parallelism cannot work. What the single model buys:
The facts, and the second one is easy to miss:
The practical consequence is about reading a stack trace: a gomp_thread_start frame with BELFEM code above it does not imply a BELFEM pragma. It is not a second source of the stack-overflow class described in doc/parallel_execution.md — that needs an OpenMP array reduction, and neither backend uses a reduction of any kind (Blaze partitions the destination, Armadillo uses critical/atomic). Third-party solvers (STRUMPACK, MKL, threaded BLAS, PARDISO) also spawn workers. BELFEM's own OpenMP touchpoints are only queries about their budget — omp_get_max_threads for the banner and the oversubscription warning below, and PARDISO's thread parameter — and those stay on the plain OMP define by design.
The sanctioned exception — solver-internal threading. Third-party solvers thread internally (STRUMPACK/SLATE, threaded BLAS). BELFEM accommodates them without becoming multithreaded itself:
The honest counterargument, kept on the table: pure MPI replicates per-rank state. On high-core-count nodes, rank-level memory replication may eventually force fewer-ranks-with-threads — that decision will be made from measurements on the memory-bound configuration, not from folklore in either direction. (Earlier revisions carried specific efficiency percentages for MPI vs. hybrid; they were unmeasured and are gone. See Benchmarks We Owe Ourselves.)
If you need threading around BELFEM today: serialize the calls —
or better, use finer MPI decomposition (more ranks, one thread each).
1. Initialization — plain MPI_Init by default; MPI_THREAD_MULTIPLE only under STRUMPACK (cl_Communicator.cpp:100-131); the Communicator constructor then queries rank, size, and MPI_TAG_UB.
2. The rank ceiling (documented limit). Message tags encode the communicating rank pair: comm_tag(aSource, aTarget) = 2*(tMax*gComm.size()+tMin) % gComm.max_tag() (src/comm/commtools.cpp:64-73), with base tag for the size header and base+1 for the payload. From MPI_TAG_UB the constructor derives and enforces a hard rank ceiling tMaxRank = 0.5*(sqrt(2*mMaxTag+1)+1) (src/comm/cl_Communicator.cpp:169-187): about 32,768 ranks at the most generous MPI_TAG_UB (INT_MAX), less on implementations reporting a smaller tag bound. Until the tag scheme is redesigned, that is BELFEM's scaling limit, and this document makes no core-count claims beyond it. The redesign is tractable: the scheme is fully encapsulated in commtools.hpp (no public API takes a tag; tags are recomputed symmetrically on both sides), so per-pair sequence counters or per-phase communicators are a comm-layer-local change — tracked as a proposal in the appendix.
3. Rank-specific execution:
4. Global variable synchronization:
5. Profiler generates rank-specific files (profile.4.2.log for size=4, rank=2) so each rank's profile is analyzed separately.
Earlier revisions of this document carried performance numbers (cache-penalty factors, alignment throughput gains, MPI-vs-hybrid efficiency percentages) that no BELFEM measurement supports. They have been removed or replaced with cited literature heuristics. The following measurements would let future revisions state them as facts:
| Claim to substantiate | Measurement needed |
|---|---|
| Cost of wrong-order matrix iteration | Micro-benchmark: row-inner vs column-inner fill on release build, sizes spanning L1/L2/LLC |
| Value of SIMD-aligned numerical buffers | Same BLAS-heavy kernel on Blaze-AVX2 (32 B) vs AVX-512 (64 B) builds |
| Pure-MPI parallel efficiency | Strong+weak scaling study of a representative h-φ problem; report efficiency vs. rank count up to the tag-scheme ceiling |
| MPI_THREAD_MULTIPLE init cost | Same run, STRUMPACK build vs plain build, on the production MPI stack |
| Release-vs-debug check overhead | make check workload timed under USE_DEBUG=ON vs OFF |
Until a benchmarks/ suite exists, treat any performance number outside this document's cited heuristics as unverified.
| ✅ | What to Remember |
|---|---|
| Naming | Prefix variables (a = argument, t = temporary, m = member, g = global). Relax for pure math. |
| Containers | Cell<T> for arrays, Vector/Matrix for linear algebra, DynamicBitset for large boolean sets. Size accessor: size() on Cell/DynamicBitset, length() on Vector/Matrix. |
| Memory | Allocate deliberately at setup. Plain T* = non-owning borrow; owners delete in their destructors; Rule of Five on every raw-owning class. Smart pointers OK as setup-scope lifetime handles. |
| Performance | Profile before optimizing. No hidden allocations in loops; scratch objects live as members. |
| Error Handling | BELFEM_ASSERT = "BELFEM has a bug" (debug-only); BELFEM_ERROR = "can fail in a correct program" (always). Once-per-run code defaults to BELFEM_ERROR. Convergence failure returns status, never aborts. |
| Thread Safety | Core is not thread-safe. One parallel model: pure MPI. Solver-internal threading is the sanctioned exception. |
| MPI | Broadcast small payloads; share/receive (with rank guard!) for large Vector<T>. Chunking is in elements of T (gMaxCommChunkLength = 65 536 elements). |
| Alignment | Comes from the backend (posix_memalign, SIMD-width dependent). Do not hand-roll aligned allocation without the C11 size-rounding and return-code rules. |
| Builds | USE_DEBUG=OFF = -O2 + NDEBUG; USE_DEBUG=ON = -Og + assertions. The default follows the SCLS flavor: OFF with no $SCLS or the gcc/mkl flavors, ON for the debug flavor. Exceptions are currently enabled in all builds; don't rely on throw in library code anyway. |
Use Cell<T> when:
Use Vector<T> / Matrix<T> when:
Use DynamicBitset when:
Use ShiftRegister<T> when:
Use manual memory when:
Use smart pointers when:
On Clean Code: BELFEM borrows exactly three things — the boy-scout rule, intention-revealing names, and comments that explain why (practiced here as equation citations). Its small-method, OO-heavy decomposition style is deliberately not adopted: scattering a numerical kernel across a dozen tiny methods destroys the correspondence between code and published equations that Mathematical Readability exists to protect.
This document has been through two source-reconciliation passes (2026-06-14 B1-scoped; 2026-08-08 full). The complete per-claim ledgers — verdict, old text, new text, citation — live in tmp/whitepaper/coding_philosophy_CHANGES.md. Summary of what the 2026-08-08 pass changed:
Corrected against source:
Removed (unverifiable performance figures): the 8×/2-3×/30-40%/10-30%/>95%/70-85%/95-vs-75 numbers and the ">100,000 cores" claim (superseded by the enforced ~32,768-rank tag ceiling, now documented as a limit). What remains cites Drepper/Agner Fog as literature heuristics.
Defects surfaced by the 2026-08-08 verification — all fixed 2026-08-09 (details in the CHANGES ledger; three-AI jury round in tmp/ai_exchange/review_matrix_mpi_and_test_gating.md):
Pending executable gate: an MPI make check run including the shrink-then-transfer regression test (source-trace evidence only until then).
Possible future directions (explicitly out of scope before 1.0): a pooled/SoA element store following the ShiftRegister placement-new discipline; the tag-scheme redesign; snippet-extraction CI so this document's code blocks can never drift from source again.
Maintained by Christian Messe, Lawrence Berkeley National Laboratory.
Any deviation from these principles should be documented in a GitHub issue with performance justification.
1.18.0