Test Design and Results

Test Design and Results#

This document describes the test design, main test cases, and most recent local results for the current tests/ suite. The suite focuses on numerical agreement between the new Torch-first/Pythonic APIs and the legacy APIs, package export compatibility, device and dtype preservation, static dependency constraints, and CPU/GPU timing samples.

Test Run Metadata#

Field

Value

Test date

2026-07-11

Package version under test

0.2.1

Legacy comparison baseline

0.1.0 API behavior retained in legacy modules

Command

python -m unittest discover -s tests -p "test_*.py" -v

CUDA availability

Not available in the most recent run

Version Comparison#

The current tests are designed around a compatibility comparison between the legacy 0.1.0 API behavior and the new 0.2.1 API surface.

Area

0.1.0 Legacy API

0.2.1 New API

Test coverage

Core MNN math

fast_dawson.py, mnn_utils.py, mnn_pytorch.py; NumPy/SciPy-backed CPU paths appear in core calculations

torch_dawson.py, torch_core.py, torch_activation.py; Torch-first Tensor paths

Numerical forward/backward agreement, autograd gradients, CPU/GPU timing hooks

NN layers

Legacy names such as OriginMnnActivation, LinearDuo, CustomBatchNorm1D

Pythonic names such as MomentActivation, MomentLinear, CustomMomentBatchNorm1d, MomentBlock

Layer-by-layer output/gradient agreement and static dependency checks

Models

Legacy MLP/ANN/SNN-style wrappers import legacy NN components

MomentMlp, MomentRateMlp, SpikeMomentMlp, AnnMlpTorch, CnnWithPoolingClassifier use new Torch-first NN APIs

Forward agreement, export checks, CPU benchmark, CUDA device-preservation check

SNN

Legacy simulation and conversion helpers, with Loihi path kept as hardware boundary

Torch-first current sources, neurons, monitors/probes, validators, and MNN-to-SNN conversion helpers

Shape/statistics agreement, device/dtype checks, conversion smoke tests

Training tools and conversion helpers

Public helpers split across general_prepare.py, functional.py, and general_train.py

Single-file mnn.utils.training_tools_api entrypoint plus mnn.utils.torch_legacy conversion helpers

Config/helper parity, minimal dataloader/optimizer/criterion setup, Torch-to-legacy MLP conversion, training-loop smoke test

How to Run#

Use the project mnn conda environment:

conda activate mnn
python -m unittest discover -s tests -p "test_*.py" -v

The most recent run did not detect CUDA, so CUDA-specific tests were skipped through unittest.skipUnless(torch.cuda.is_available()).

Result Summary#

Most recent full run:

Ran 75 tests in 0.479s

OK (skipped=5)

The skipped tests were all CUDA-only cases:

  • test_gpu_torch_activation_timing_against_vanilla_cpu

  • test_gpu_torch_core_timing_against_vanilla_cpu

  • test_gpu_torch_dawson_timing_against_vanilla_cpu

  • test_cuda_model_forward_stays_on_gpu

  • test_correlated_gaussian_current_stays_on_cuda

test_torch_mnn_core.py#

This file validates mathematical agreement between the new core modules torch_dawson.py, torch_core.py, torch_activation.py and the legacy CPU/NumPy/SciPy path. It also records timing samples for the Torch implementation on CPU/GPU against vanilla CPU.

Dawson Accuracy#

Cases:

  • test_first_order_dawson_matches_vanilla

  • test_second_order_dawson_matches_vanilla

Design:

  • Builds 129 float64 sample points on [-8, 8].

  • Uses legacy Dawson1.dawson1, Dawson1.int_fast, Dawson2.dawson2, and Dawson2.int_fast.

  • Uses new DawsonFirstOrder.evaluate(), DawsonFirstOrder.integral(), DawsonSecondOrder.evaluate(), and DawsonSecondOrder.integral().

  • Compares maximum absolute error and maximum relative error. Some integral and second-order Dawson checks primarily rely on relative error because the approximations can produce very large intermediate magnitudes.

Coverage intent:

  • Verifies that the fixed Chebyshev coefficients in the new modules align with legacy output.

  • Verifies that the torch.Tensor input path does not depend on NumPy/SciPy computation.

Core Forward/Backward Accuracy#

Cases:

  • test_compute_bounds_matches_vanilla

  • test_forward_methods_match_vanilla

  • test_backward_methods_match_vanilla

Design:

  • Builds two-dimensional mean/std tensors that include zero-standard-deviation, low-mean, and high-mean branches.

  • Compares against legacy Mnn_Core_Func.compute_bound, forward_fast_*, backward_fast_*, fast_forward, and fast_backward.

  • Maps those calls to new MNNCore.compute_bounds(), forward_mean(), forward_std(), forward_chi(), backward_mean(), backward_std(), backward_chi(), forward(), and backward().

  • Additionally checks that the compute_bounds mask exactly matches the legacy mask.

Coverage intent:

  • Verifies compatibility across piecewise branches, masks, zero-variance boundaries, and aggregate forward/backward interfaces.

  • Ensures the new function decomposition does not change the legacy formulas.

Activation Accuracy and Autograd#

Cases:

  • test_activation_without_correlation_matches_vanilla_forward_and_backward

  • test_activation_with_correlation_matches_vanilla_forward_and_backward

  • test_activation_float32_cutoff_region_uses_stable_internal_precision

Design:

  • Compares legacy mnn_activate_no_rho with new mnn_activation_without_correlation.

  • Compares legacy mnn_activate_trio with new mnn_activation_with_correlation.

  • Compares every forward output tensor.

  • Sums all outputs and runs backward, then compares gradients for mean, std, and correlation.

  • Exercises a float32 cutoff-region input where the activation internally promotes calculations to float64, then verifies finite forward outputs, finite gradients, and float32 output dtype preservation.

Coverage intent:

  • Verifies that the new Torch-only custom autograd path matches legacy forward and backward behavior.

  • Ensures correlation-matrix diagonal handling keeps the legacy behavior.

Benchmarks#

Cases:

  • test_cpu_torch_dawson_timing_against_vanilla_cpu

  • test_cpu_torch_core_timing_against_vanilla_cpu

  • test_cpu_torch_activation_timing_against_vanilla_cpu

  • CUDA counterparts when CUDA is available

Design:

  • Uses 2048 float64 points and averages over 10 repeats.

  • CPU tests compare vanilla CPU and Torch CPU.

  • GPU tests compare vanilla CPU and Torch CUDA, synchronizing CUDA after each call.

  • Benchmark tests still assert numerical agreement, so they do not measure speed without correctness.

Most recent CPU results:

Case

Vanilla CPU

Torch CPU

activation_without_correlation

0.000463s/call

0.001432s/call

activation_with_correlation

0.003504s/call

0.005781s/call

nn.MomentActivation

N/A

0.013216s/call

core.forward

0.000472s/call

0.001446s/call

core.backward

0.000616s/call

0.003563s/call

dawson_first.evaluate

0.000018s/call

0.000017s/call

dawson_first.integral

0.000166s/call

0.000421s/call

dawson_second.evaluate

0.000136s/call

0.000572s/call

dawson_second.integral

0.000120s/call

0.000587s/call

Notes:

  • On the current CPU, the Torch implementation is usually slower than the NumPy/SciPy legacy path, but it preserves Tensor, dtype, and GPU semantics.

  • dawson_first.evaluate was slightly faster in Torch in this CPU sample.

  • GPU timing should be refreshed in a CUDA-enabled environment.

test_torch_nn_api.py#

This file validates the new Pythonic Torch API under mnn.mnn_core.nn, including package exports, numerical agreement with legacy layers, and best-practice structural constraints.

Import and Static Dependency Tests#

Cases:

  • test_new_api_exports_from_nn_package

  • test_new_modules_do_not_import_legacy_core_modules

Design:

  • Verifies that package-level exports in mnn.mnn_core.nn point to the new classes, including MomentActivation, MomentLinear, MomentBatchNorm1d, CustomMomentBatchNorm1d, MomentBlock, and MomentPooling.

  • Uses inspect.getsource to statically check that new *_torch.py modules do not contain mnn_pytorch, mnn_utils, or fast_dawson.

Coverage intent:

  • Ensures the explicit new API is importable from the package entrypoint.

  • Prevents the new NN layers from routing back through legacy core implementations.

Layer Accuracy Tests#

Cases:

  • test_activation_matches_legacy_forward_and_backward

  • test_constant_current_activation_matches_legacy

  • test_linear_matches_legacy

  • test_linear_no_correlation_matches_legacy

  • test_batch_norm_matches_legacy_eval

  • test_batch_norm_no_correlation_matches_legacy_eval

  • test_custom_batch_norm_matches_legacy_eval

  • test_pooling_matches_legacy

  • test_criterion_matches_legacy

  • test_ensemble_matches_legacy_eval

  • test_ensemble_no_correlation_matches_legacy_eval

Design:

  • Builds float64 mean/covariance/std inputs.

  • Instantiates legacy and new modules, then copies legacy weights, biases, running statistics, and variance-bias parameters into the new modules under torch.no_grad().

  • Uses eval-mode batch norm and ensemble tests to avoid training-statistics differences.

  • Activation tests also compare backward gradients.

Coverage intent:

  • Verifies that new Pythonic class names and internal implementations preserve legacy layer mathematics.

  • Covers both correlation and no-correlation paths.

  • Covers loss, pooling, batch norm, linear, activation, and ensemble layer behavior.

Best-Practice Tests#

Cases:

  • test_sparse_mask_is_buffer_and_uses_torch_shape

  • test_moment_conv_uses_module_list

  • test_moment_conv_forward_shapes

Design:

  • Checks that sparse_mask is registered as a buffer, has the same shape as weight, and has the expected number of sparse connections per output neuron.

  • Checks that convolution submodules are registered through torch.nn.ModuleList, so PyTorch can track their parameters.

  • Checks mean and covariance output shapes for convolution forward.

Coverage intent:

  • Guards against unregistered parameters/buffers, NumPy-based sparse masks, and shape mismatches in the new implementation.

test_torch_models_api.py#

This file validates the new Torch-first model API in mnn.models, including MLP, rate-only, spike-style, and ANN wrappers.

Export and Static Dependency Tests#

Cases:

  • test_exports_new_model_api

  • test_new_model_sources_do_not_use_legacy_nn_api

Design:

  • Verifies package-level exports for MomentMlp, SpikeMomentMlp, MomentRateMlp, and AnnMlpTorch.

  • Statically checks that models.mlp_torch and models.cnn_torch do not reference legacy NN names: LinearDuo, OriginMnnActivation, CustomBatchNorm1D, or legacy mnn_core.nn.functional.

Coverage intent:

  • Ensures the new models API depends on new NN layers rather than indirectly reusing legacy implementations.

Forward Compatibility Tests#

Cases:

  • test_moment_mlp_matches_legacy_forward

  • test_spike_moment_mlp_matches_legacy_forward

  • test_rate_mlp_matches_legacy_forward

  • test_ann_mlp_torch_shape_matches_legacy

Design:

  • Copies legacy model weights and batch norm state into the new MNN, SNN-style, and mean-only/rate MLP models.

  • Uses fixed-seed float64 mean/covariance inputs or ordinary Tensor inputs.

  • Compares forward outputs numerically or by shape, depending on the model type.

Coverage intent:

  • Confirms that the new model construction preserves legacy model composition semantics.

  • Keeps the ANN wrapper check scoped to output shape, preventing ordinary Tensor models from accidentally adopting moment-tuple semantics.

Benchmarks and CUDA#

Cases:

  • test_cpu_benchmark_model_api

  • test_cuda_model_forward_stays_on_gpu

Design:

  • CPU benchmark compares legacy MnnMlp and new MomentMlp, averaging over 3 repeats.

  • CUDA test verifies that MomentMlp inputs and outputs stay on GPU when CUDA is available.

Most recent CPU results:

Case

Time

legacy CPU MnnMlp

0.000454s/call

Torch CPU MomentMlp

0.001270s/call

test_torch_snn_api.py#

This file validates the new Torch-first API under mnn.snn and mnn.snn.base, including current sources, Poisson spike sources, LIF neurons, monitors/probes, top-level functional helpers, and MNN-to-SNN conversion.

Export and Static Dependency Tests#

Cases:

  • test_exports_new_snn_api

  • test_new_snn_sources_do_not_use_numpy_or_legacy_core

  • test_top_level_functional_torch_exports

Design:

  • Verifies package-level exports for new APIs such as LifNeurons, SpikeMonitorTorch, GaussianCurrentSource, PoissonSpikeSource, and MomentMlpToSnn.

  • Statically checks that new SNN Torch modules do not contain import numpy, import scipy, .numpy(), from_numpy, .data, or legacy core dependencies.

  • Verifies that snn.functional_torch validators and helpers are also exported at the top level.

Coverage intent:

  • Ensures the core calculation path of the new SNN implementation is Torch-first.

  • Prevents hidden CPU/NumPy fallback in simulation or conversion code.

Functional Helper Tests#

Cases:

  • test_sample_shape_matches_legacy_sample_size

  • test_top_level_poisson_sample_matches_legacy_shape

  • test_top_level_sparse_statistics_matches_legacy

  • test_condition_modifier_uses_torch_correlation

Design:

  • Checks sample_shape against legacy sample-size semantics.

  • Checks Poisson sampling shape and dtype against the legacy helper.

  • Compares sparse spike train statistics with legacy sparse_spike_train_statistics.

  • Checks that MomentSnnValidator.modify_value_by_condition zeros the mean and preserves covariance diagonals under the corr_only condition.

Coverage intent:

  • Covers the parallel refactor of the top-level snn/functional.py API.

  • Confirms that statistical helpers and condition-modification logic preserve legacy semantics.

Current Source and Device Tests#

Cases:

  • test_independent_gaussian_current_shape_dtype_device

  • test_correlated_gaussian_current_uses_torch_device

  • test_correlated_gaussian_current_stays_on_cuda

  • test_poisson_source_shape_matches_legacy

Design:

  • Independent Gaussian current checks output shape, dtype, and device.

  • Correlated Gaussian current uses the covariance/Cholesky path and checks pregenerated output shape (num_steps, *shape).

  • CUDA test verifies that outputs stay on GPU when CUDA is available.

  • Poisson spike source compares output shape against the legacy generator.

Coverage intent:

  • Verifies that random current sources do not fall back to CPU and preserve the dtype/device of their Tensor inputs.

  • Uses shape/statistical-interface checks for legacy random semantics rather than requiring sample-by-sample equality.

Neuron, Monitor, Probe, and Conversion Tests#

Cases:

  • test_lif_neurons_matches_legacy_single_step

  • test_monitor_and_probe_behavior

  • test_convert_moment_parameters_and_snn_forward

  • test_spike_moment_to_snn_forward

Design:

  • Compares one-step LIF output with legacy LIFNeurons, and checks reset state alignment.

  • Compares spike monitor count and duration.

  • Compares probe record keys and basic export behavior; probe/export is treated as an explicit CPU collection boundary rather than a hidden computation fallback.

  • Checks MNN-to-SNN conversion parameter keys and validates converted forward output shape.

Coverage intent:

  • Covers simulation state, recorders, and new model conversion paths.

  • Confirms that conversion APIs can identify new models/layers and produce runnable SNN forward paths.

Benchmark#

Case:

  • test_cpu_benchmark_snn_current

Most recent CPU results:

Case

Time

legacy CPU GaussianCurrentGenerator

0.000014s/call

Torch CPU GaussianCurrentSource

0.000008s/call

test_training_tools_api.py#

This file validates the new single-file training tools entrypoint mnn.utils.training_tools_api while ensuring the legacy mnn.utils.training_tools package still works.

Export and Static Dependency Tests#

Cases:

  • test_exports_and_legacy_package_still_work

  • test_source_does_not_import_legacy_training_modules

Design:

  • Verifies that the new API exposes TrainProcessCollections.

  • Verifies that legacy package-level exports still point to objects from legacy general_prepare, functional, and general_train.

  • Statically checks that the new single-file API does not proxy old modules through from . import general_prepare, from . import functional, or from . import general_train.

Coverage intent:

  • Ensures the new single-file API is an integrated implementation rather than a facade over the old modules.

  • Ensures legacy entrypoints remain available.

Config and Helper Tests#

Cases:

  • test_config_read_and_reset_match_legacy

  • test_transform_compose_matches_legacy_type

  • test_input_predict_score_helpers

  • test_average_and_progress_meter

Design:

  • Uses a temporary YAML file to compare set_config2args behavior against legacy behavior, including special-argument reset.

  • Checks that transform composition returns the same object type as the legacy helper.

  • Checks InputPreprocess dtype conversion and mean masking.

  • Checks argmax prediction and equality-based scoring helpers.

  • Checks AverageMeter accumulation and ProgressMeter display text.

Coverage intent:

  • Covers the core training-flow utilities for configuration, input preprocessing, prediction, scoring, and logging display.

Torch-to-Legacy MLP Conversion Tests#

Cases:

  • test_build_legacy_mlp_from_moment_mlp_matches_forward

  • test_build_legacy_no_correlation_mlp_matches_forward

  • test_build_legacy_rate_mlp_matches_forward

  • test_copy_torch_mlp_to_legacy_returns_target

  • test_sparse_mask_is_copied_to_legacy_mlp

  • test_copy_torch_mlp_to_legacy_rejects_unsupported_pair

  • test_copy_torch_mlp_to_legacy_rejects_shape_mismatch

Design:

  • Builds legacy models from MomentMlp, MomentMlpNoCorrelation, and MomentRateMlp, then compares their forward outputs with the source models on fixed inputs.

  • Uses mnn.utils.torch_legacy as the canonical conversion module and checks that training_tools_api re-exports the same helpers.

  • Checks that direct copy returns the passed target model.

  • Checks that sparse masks are preserved when converting sparse MomentMlp instances.

  • Checks error handling for unsupported model pairs and strict shape mismatches.

Coverage intent:

  • Verifies that copy_torch_mlp_to_legacy and build_legacy_mlp_from_torch can bridge new MLP checkpoints back into equivalent legacy models for migration, debugging, and comparison workflows.

Training Preparation and Pipeline Smoke Tests#

Cases:

  • test_dataloader_optimizer_and_criterion_smoke

  • test_train_and_validate_smoke

Design:

  • Uses a tiny TensorDataset to avoid external dataset downloads.

  • Verifies minimal usable paths for make_dataloader, prepare_optimizer_scheduler, and make_criterion.

  • Runs TrainProcessCollections.train_one_epoch and validate with torch.nn.Linear, CrossEntropyLoss, and SGD.

Coverage intent:

  • Verifies that the new single-file API can run a minimal training loop.

  • Keeps training-flow coverage as a smoke test rather than tying it to real dataset downloads or long training jobs.

Torch-First Construction and Configuration Tests#

Cases:

  • test_make_model_defaults_to_torch_mlp_with_legacy_yaml_fields

  • test_make_mnn_mlp_model_selects_each_torch_family

  • test_make_model_can_explicitly_request_legacy_implementation

  • test_config_mnn_activation_configures_torch_core_only

  • test_make_criterion_defaults_to_torch_mnn_criterion

Design:

  • Uses an MNIST-style legacy YAML model mapping and verifies that make_model() constructs MomentMlp, while translating legacy argument names to the Torch-first attributes.

  • Checks all supported MLP model-family selectors: moment, spike moment, no-correlation, rate, and ANN.

  • Verifies the opt-in meta.use_legacy_api: true path still returns MnnMlp for migration and checkpoint comparison.

  • Changes activation settings through config_mnn_activation() and checks the Torch core changes without changing the legacy core singleton.

  • Requests CrossEntropyOnMean from source: mnn_core and checks that the returned module is CrossEntropyOnMeanTorch; it also checks legacy criterion argument aliases with MSEOnMean.

Coverage intent:

  • Ensures the new training entrypoint no longer silently routes normal model construction, activation configuration, or MNN loss selection through legacy implementations.

  • Preserves existing YAML configuration files as migration inputs while keeping the Torch-first API as the default execution path.

Docs Index Test#

Case:

  • test_docs_migration_files_are_indexed

Design:

  • Checks that docs/index.rst includes legacy_nn_migration, legacy_models_snn_migration, and legacy_training_tools_migration.

  • Checks that the corresponding Markdown files exist.

Coverage intent:

  • Ensures newly added migration docs are not omitted from the documentation index.

Coverage Boundaries#

Currently covered:

  • New-vs-legacy core mathematical outputs and gradients.

  • Explicit exports for new NN layers, models, SNN APIs, and training tools.

  • Static dependency constraints for new Torch-first modules.

  • CPU benchmarks and GPU device-preservation checks when CUDA is available.

  • Minimal training-loop smoke coverage.

Not covered, or only lightly covered:

  • CUDA benchmarks were not run on the current machine and should be refreshed in a CUDA-enabled environment.

  • Random SNN sampling mainly checks shape, dtype, device, and statistical helpers, not sample-by-sample equality.

  • The ANN wrapper currently checks output shape only.

  • Training tools do not run real dataset downloads, distributed training, multiple scheduler argument combinations, or full checkpoint resume workflows.

  • Documentation build was not run in this command; documentation references are checked through unittest index assertions only.