TEM build scripts

Inside the build container (Deploy the build container) the lab sequence is three wrappers. Each one is a single nornir-build process. Stages run in order, joined by --then, and the process stops on the first failure. The default compute library is CuPy (NORNIR_COMPUTATIONAL_LIBRARY). Progress is published to the co-located dashboard.

Run them in this order:

  • TEMImport when the volume has not been imported.

  • TEMBuild to build mosaics. It assumes VolumeData.xml already exists. TEMBuild-import is import plus the same mosaic stages.

  • TEMAlign after the mosaics exist. It produces the slice-to-volume transforms Viking reads.

The science for these stages is Overview of volume building. Flags below are the ones in nornir-buildmanager/scripts. nornir-build <Pipeline> --help lists every flag; the pipeline catalogue (Pipeline catalogue) is not this recipe.

TEMImport

ImportIDoc copies a SerialEM .idoc directory into the volume and writes VolumeData.xml at the volume, block, section, and channel levels.

TEMImport /path/to/idoc /path/to/volume

TEMBuild

Same chain as the VS Code launch configuration “TEM Build: full sequence (sequential)”. Pass the volume directory, or let the script prompt.

  • Prune -InputFilter Raw8 -Downsample 4 -Channels TEM -DefaultThreshold 10.0 — drop tiles that do not have enough texture to align.

  • Histogram -Filters Raw8 -InputTransform Prune -Downsample 4 -Channels TEM — contrast statistics for the tiles that remain.

  • AdjustContrast -InputFilter Raw8 -OutputFilter Leveled -InputTransform Prune -Channels TEM — write the Leveled filter that mosaic registration uses.

  • Mosaic -InputFilter Leveled -RegistrationDownsample 4 -InputTransform Prune -OutputTransform Grid -Channels TEM — lay the tiles out. The output transform is Grid.

  • Assemble -Channels TEM -Filters Leveled -Downsample 8,16,32 -NoInterlace -Transform Grid — build pyramid images from that layout.

  • MosaicReport -PruneFilter Raw8 -ContrastFilter Raw8 -AssembleFilter Leveled -AssembleDownsample 16 -Output MosaicReport — QA report for the mosaic.

  • CreateVikingXML -OutputFile Mosaic — a mosaic manifest. This is not a slice-to-volume group yet.

TEMAlign

Run this after TEMBuild. A bad brute-force pair is fixed in Pyre (Using Pyre) before you treat the later grid passes as final.

  • CreateBlobFilter -Channels TEM -InputFilter Leveled -Levels 16,32,64 -OutputFilter Blob -Radius 9 -Median 7 -Max 3 — a texture image so downsampled brute force still sees features.

  • AlignSections -NumAdjacentSections 1 -Filters Blob -UseMasks -Downsample 64 -Channels TEM — first slice-to-slice placement, one neighbor each way, and a potential registration chain.

  • AssembleStosOverlays -StosGroup StosBrute -Downsample 64 -StosMap PotentialRegistrationChain — overlay pictures of those pairs.

  • SelectBestRegistrationChain -StosGroup StosBrute -Downsample 64 -InputStosMap PotentialRegistrationChain -OutputStosMap FinalStosMap — choose the chain and write FinalStosMap.

  • RefineSectionAlignment -InputGroup StosBrute -InputDownsample 64 -OutputGroup Grid -OutputDownsample 32 -Filter Leveled — first grid refine, from the brute result at 64 to a grid at 32.

  • AssembleStosOverlays -StosGroup Grid -Downsample 32 -StosMap FinalStosMap — overlays of that grid.

  • CreateVikingXML -StosGroup Grid32 -StosMap FinalStosMap -OutputFile Grid32 — publish that pass. Grid32 is a StosGroup name Viking can show.

  • RefineSectionAlignment -InputGroup Grid -InputDownsample 32 -OutputGroup Grid -OutputDownsample 16 -Filter Leveled — second grid refine, 32 to 16.

  • SliceToVolume -Downsample 16 -InputGroup Grid -OutputGroup SliceToVolume -NoLinearBlend — one mapping per section onto the center section.

  • ScaleVolumeTransforms -InputGroup SliceToVolume -InputDownsample 16 -OutputDownsample 1 — scale that mapping to full resolution.

  • LinearizeVolume -InputGroup SliceToVolume -InputDownsample 1 -OutputGroup SliceToVolumeLinear -min_blend 0.005 -max_blend 0.05 -travel_limit 512 -reblend_iterations 8 -reblend_tolerance 0.5 — a blended variant of the same mapping.

  • CreateVikingXML -OutputFile SliceToVolume -StosGroup SliceToVolume1 -StosGroup SliceToVolumeLinear1 -StosMap SliceToVolume — the Viking manifest. Those StosGroup names are the labels in Viking’s File → Transform → SliceToVolume menu.

  • MosaicToVolume -InputTransform Grid -OutputTransform ChannelToVolume -Channels '(?!Registered)' — map each mosaic channel into volume space.

  • Assemble -Channels '(?!Registered)' -Filters Leveled -Downsample 32 -NoInterlace -Transform ChannelToVolume -ChannelPrefix Registered_ — write Registered_* channels at downsample 32.

  • MosaicReport -PruneFilter Raw8 -ContrastFilter Raw8 -AssembleFilter Leveled -AssembleDownsample 32 -Output VolumeReport — QA report for the volume.

  • ExportImages -Channels Registered -Filters Leveled -Downsample 32 — export the registered leveled images next to the volume.

View and annotate the result in Viking. Analyze those annotations in SBFSEM-tools.