> For the complete documentation index, see [llms.txt](https://gaps.gitbook.io/gaps/jQBaAQpYuk4HtfLcsK7Y/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://gaps.gitbook.io/gaps/jQBaAQpYuk4HtfLcsK7Y/m4.md). # Module 4 - Reproducibility [← Previous module](/gaps/jQBaAQpYuk4HtfLcsK7Y/m3.md) | [GAPS structure](/gaps/jQBaAQpYuk4HtfLcsK7Y/structure.md) | [Next module →](/gaps/jQBaAQpYuk4HtfLcsK7Y/m5.md) *** ![](/files/reb54ujNy9msrNMdzxeF) ## Module 4 - Reproducibility Reproducibility is not a single property. It is a combination of dimensions that determine how easily others can understand, execute, verify, and build upon your work. Publicly sharing code and data alone is often insufficient. Missing environment information, undocumented workflows, manual processing steps, and unclear links between artifacts and publications may still prevent others from validating or extending the research. ## Learning objectives By the end of this module, you will be able to: * Explain key dimensions of artifact reproducibility and how they support independent verification and reuse. * Apply reproducibility practices to support independent verification of research workflows and outputs. * Apply version control practices to track changes and support collaboration throughout the research lifecycle. * Validate and curate artifacts before release to ensure completeness, consistency, and independent execution. * Document result variability, known limitations, and justifiable deviations from exact reproduction. ## Why reproducibility matters > **Key principle:** Reproducible artifacts should allow external users to understand, execute, and validate the reported results without relying on undocumented knowledge from the original authors. Three non-negotiable expectations for reproducible artifacts: * **Connect to the paper.** Scripts, datasets, workflows, and outputs should be explicitly linked to the specific claims, figures, tables, and analyses in the publication. * **Work without author involvement.** Reviewers and future users should be able to retrieve, install, execute, and understand the artifact independently. * **Provide sufficient experimental transparency.** Others must be able to understand how conclusions were obtained and assess whether the evidence supports the reported findings. ## Dimensions of reproducibility The table below summarizes key dimensions of artifact reproducibility and illustrates how artifacts evolve from limited usability to fully reproducible and durable research assets. | Dimension | Limited artifact | Operational artifact | Durable artifact | | ----------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | **Accessibility** | No or fragmented artifacts; links missing, private, or ephemeral. | Artifacts hosted but with fragile links (personal cloud, ad-hoc URLs); partial coverage of code, data, and models. | Artifacts versioned and persistently hosted (e.g., PID, archival repository) with clear mapping to experiments. | | **Environment specification** | Missing or incomplete environment requirements. | Basic installation steps documented, but incomplete dependency lists and implicit platform assumptions. | Complete, machine-readable environment specification (e.g., lockfiles, container specs) covering OS, libraries, and hardware. | | **Versioning rigor** | Floating or unspecified versions. | Some versions documented (e.g., major library versions) but not consistently pinned. | Pinned versions at all hierarchy levels (datasets, models, frameworks, OS), with change logs or manifests. | | **Execution fidelity** | No runnable scripts; only high-level prose or pseudo-code. | Pipelines runnable with manual effort (e.g., several shell commands, manual downloads, ad hoc scripts). | End-to-end, automated pipelines or containers that recreate results from scratch with minimal manual steps. | | **Legal openness** | Licenses absent, incompatible, or ambiguous. | Some components licensed, but coverage incomplete. | Explicit, compatible licenses for all artifacts required to reproduce results. | ## Automation for reproducibility > **Key principle:** Reproducing results manually is error-prone, difficult to verify, and hard to maintain. Automation makes workflows traceable, repeatable, and independently executable. The scripted workflow principles and Makefile-based orchestration introduced in [Module 2 - Preparation](/gaps/jQBaAQpYuk4HtfLcsK7Y/m2.md) form the technical backbone of reproducibility. This section expands on them with a focus on robustness and independent execution. ### End-to-end automation > **Goal:** Allow the complete workflow (from raw inputs to final outputs) to be executed with a single command or a minimal sequence of steps. * Provide a **single command or script** that runs the full experiment. * Provide a **single script** that generates all relevant figures and results from the paper. * Automate build, evaluation, and visualization pipelines. * Write **parameterized scripts** with command-line flags so users can customize execution without modifying source code. * Include **progress messages or estimated duration** for each execution step. * When installation requires multiple setup steps, provide installation scripts or automated setup procedures. ### Automated figure and table generation > **Rule:** Figures, tables, and statistical summaries should be generated automatically from data and scripts, never manually edited. Manual steps create inconsistencies between the artifact and the published results. * Provide scripts to automatically regenerate all tables and figures presented in the paper. * For each command, note the output files it writes to, so users know where to find the results. * Generate data in the same format and organization as in the paper. * e.g., for a table, include a script that generates a similar table; for a plot, generate a similar plot. * Auto-generate plots directly from experimental results. Do not ask users to manually inspect log files and compare numbers with figures in the paper. ### Modular and incremental execution For complex or long-running workflows, provide modular and interruptible scripts that allow individual steps to be executed independently. * Allow pausing and resuming execution. * Support running on small input subsets. * Include **smoke tests** and quick validation steps that confirm core functionality without running the full experiment. * Provide modular scripts for individual steps of complex or long-running tasks. > **Tip:** If the artifact runs for more than a few minutes, explain how to run it on smaller inputs and how long the full run is expected to take. ## Managing dependencies and execution environments > **Key problem:** Many reproducibility failures stem not from missing scripts or data, but from undocumented or inconsistent execution environments. ### Dependency specification > **Rule:** All required dependencies must be explicitly specified with **pinned versions**. Incomplete or implicit dependency specifications are one of the most common causes of reproduction failures. The dependency specification files introduced in [Module 2 - Preparation](/gaps/jQBaAQpYuk4HtfLcsK7Y/m2.md) apply here equally. Always document the exact execution environment: * Operating system and kernel version. * Hardware requirements (CPU, GPU, memory, storage, specialized peripherals). * All software dependencies and their exact versions. * Justification for any deviation from standard environments. > **Tip:** Remove unused dependencies and installable packages before release to reduce setup complexity and avoid unnecessary installation failures. ### Environment isolation and portability * Artifacts should be as **self-contained** as possible. Avoid relying on external services, undocumented configurations, or transient resources. * Prefer **open-source tools** over proprietary or closed-source dependencies. * If proprietary tools are unavoidable, document them explicitly with instructions for how to obtain and use them. * **Avoid runtime downloads** from external services. Required datasets, models, and dependencies should be packaged with the artifact or referenced through stable archival sources. * Prefer **open and widely supported formats** for archives, datasets, and documents (e.g., ZIP, TAR.GZ, CSV, JSON, PDF). * **Avoid hardcoded paths.** Use repository-relative paths to improve portability. * Clearly document **architecture-specific requirements** (e.g., ARM vs. x86, GPU dependencies, OS constraints). ### Containers and virtual machines > **Tip:** A pre-built container or VM image is preferable to setup scripts alone. Pre-built images eliminate reliance on external dependencies during the configuration step. * Software artifacts should ideally be contained within a VM or container that includes all dependencies, reducing the likelihood of artifact decay over time. * When using Docker, ensure what you distribute is **fully self-contained**. A base container that installs dependencies at runtime reduces artifact size but increases reliance on external systems that may eventually become unavailable. * Non-software artifacts (e.g., datasets) should be distributed as a single archive. * A container is not necessary for non-executable materials. * Ship VMs in a **portable format (OVA or OVF)**. * Always provide the **Dockerfile, provisioning scripts, or initialization scripts** used to create the environment, not only the pre-built image. * Document the internal organization of containers and VMs: where source code, datasets, scripts, generated outputs, and documentation are located. > **Tip:** Simple usability improvements help significantly. For example, a terminal already opened in the main artifact directory, or shortcut scripts for common tasks. > **Tip:** For lighter-weight isolation, language-level virtual environments are a practical alternative for many types of artifacts. ### Virtualized vs. bare-metal execution Some performance-related experiments may not reproduce accurately in virtualized environments. In these cases: * Explicitly document the limitations of virtualized execution. * Document the hardware specifications used in your experiments. * If performance claims cannot be reproduced in a VM, include instructions for running benchmarks on bare metal. * When specialized hardware cannot be virtualized, provide instructions for how users can obtain access to those resources. ### Long-term dependency management Dependencies evolve. Libraries are updated, APIs change, and some tools eventually become unavailable. Choices made at publication time affect whether an artifact remains executable years later. * Be aware that some dependencies may become unavailable over time, which can compromise reproducibility. * Avoid relying on complex, proprietary, or unstable third-party components whenever a simpler or more stable alternative exists. * Minimize reliance on external services, undocumented configurations, or transient resources. * Prefer tools and languages with large, stable communities that are more likely to remain supported and executable over time. * When external resources are unavoidable, archive or mirror critical dependencies alongside the artifact whenever possible. ## Supporting independent reproduction > **Key principle:** For results to be independently verifiable, the artifact must provide sufficient context for others to understand how conclusions were derived, configure the environment correctly, and interpret the outputs they produce. ### Documenting parameters and configurations Document everything that could significantly affect results: * Parameter values and default settings. * Software and dependency versions. * Hardware specifications used in the original experiments. * Execution configurations: thread counts, memory limits, random seeds, training configurations, workload sizes, cache sizes. When artifacts involve training, tuning, calibration, or iterative design decisions, **clearly distinguish data used during development from data used for evaluation**. ### File integrity verification Provide **checksums** (e.g., SHA-256 hashes) for important files or compressed packages so users can verify the integrity of downloaded materials. This is particularly important for large datasets, pre-built binaries, or VM images. ### Handling comparative evaluations When the artifact includes comparisons against other systems: * Include a version of each compared system and instructions for reproducing the comparison numbers used in the paper. * If a compared tool crashes on a subset of inputs, note this explicitly as expected behavior. * Clearly document experimental conditions for all compared systems: configurations, optimization levels, datasets, and preprocessing steps. * Differences in configurations, hardware, or datasets between compared systems must be explicitly documented. ### Reusability beyond reproduction * Use **open-source implementations** rather than closed-source or proprietary tools. * Use **public benchmarks** that others can access and run independently. * Structure and document datasets so they can be **reused across multiple studies**, not only for the paper they originally supported. * Ensure that access to the artifact does **not depend on access to the paper**. ## Validating artifacts before release > **Key principle:** Validate the artifact under realistic conditions that simulate how external users will interact with it, not just in your own development environment. This is one of the most commonly overlooked steps in artifact preparation. Authors often test artifacts only on their own machines, where implicit dependencies and configuration steps are already in place. External validation reveals problems that would otherwise only be discovered by reviewers or future users. ### Validation steps | Validation step | What to do | | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | Full rerun from scratch | Execute the entire workflow from raw inputs to final outputs without manual intervention. | | Follow your own documentation | Read and execute the instructions exactly as written, without relying on prior knowledge of the project. | | Test in a clean environment | Run on a machine where no prior setup exists to reveal implicit dependencies. | | Test on multiple platforms | Identify platform-specific issues across operating systems or hardware configurations. | | External tester | Ask a colleague or student not involved in the artifact to execute it and report problems. Iterate until no major issues remain. | | Automated tests | Run unit tests, sanity-check scripts, or automated checks on individual components. | | Verify archived outputs | Ensure that expected outputs stored for comparison have not been modified during testing. | | Test compressed packages | Verify that archives can be extracted correctly on different operating systems. Avoid absolute paths or platform-specific compression tools. | | Test copy-paste commands | Execute each command from the README exactly as written. | > **Rule:** If total reproduction time is very long on standard hardware provide intermediate pre-computed results or a minimum working example and the expected output. > **Tip:** Vet the artifact on a clean machine to confirm it can be set up within a reasonable time frame before submitting. ### Checking integrity and completeness Before release, verify the artifact for any setup problems that may prevent proper use, such as corrupted, or missing files, VMs that do not start, or immediate crashes on the simplest example input. Verify that code artifacts execute correctly after being **downloaded**, not only after local development. ## Cleaning and curating the artifact before release > **Key principle:** A clean artifact is easier to navigate, evaluate, and reuse. Remove everything that does not directly contribute to understanding or reproducing the study. * Remove internal comments, TODOs, test scripts, unused experimental code, temporary files, backups, dead code, and outdated versions. * Remove intermediate or irrelevant result files that do not correspond to the study. * Remove unused dependencies and installable packages. * Remove material not referenced or described in the paper. * Refactor code before sharing to improve readability and maintainability. * Harmonize data representation: align variable names, metrics, units, and outputs with those used in the paper. * Review data to ensure correctness before publication. > **Warning:** Do not release partial artifacts. Incomplete artifacts hinder reproducibility and fair comparison, and may prevent proper evaluation against prior work. > **Tip:** Preparing artifacts collaboratively reduces the risk of missing issues that a fresh perspective would catch. ## Documenting result variability and limitations > **Key principle:** Exact reproduction is not always achievable. Transparently documenting expected variability and justifiable deviations is more valuable than overstating reproducibility claims. ### Expected result variability For experiments involving nondeterministic behavior or performance variability, document: * The number of executions performed. * Whether warm-up phases were used. * How measurements were aggregated. * How variability was analyzed. * What level of variation is considered acceptable (replication tolerance). > **Tip:** Include measures of variability (e.g., standard deviation, confidence intervals, ranges across repeated executions) rather than only central tendency measures. Provide **archived expected outputs** in a separate folder so users can compare their results against a reference without accidentally overwriting it. > **Warning:** When using simplified experiments or toy examples, the documentation should clarify how the provided evaluation relates to real-world scenarios when the artifact claims benefits for realistic applications. ### Justifiable deviations Deviations from exact reproduction are acceptable when properly documented: | Situation | What to do | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Results are performance data and hardware-dependent. | Explain how to recognize when experiments on other hardware reproduce the high-level trends, not necessarily exact numbers. | | Evaluation takes a very long time. | Provide small and representative inputs that demonstrate the core behavior. | | Evaluation requires specialized hardware. | Provide instructions for how users can obtain access to the required hardware. | | Benchmark code is proprietary or licensed. | Include public benchmarks instead. If all major benchmarks for a key claim are private, provide alternative data. | | Experiments run for multiple days. | Provide reduced input datasets, allow partial reproduction, or provide a screencast as an alternative. | > **Tip:** Known deviations from results presented in the paper should be explicitly outlined. This includes cases where a table or figure is not produced, or where reproduced results differ from those in the paper. ### Disclosure statement Include an explicit disclosure statement in the artifact package covering: * Potential conflicts of interest or funding sources relevant to the artifact. * Limitations or constraints of the artifact. * Ethical considerations (e.g., data handling conditions, consent agreements). * A clear description of what is included and excluded from the shared materials, and why. ### Reproducibility in qualitative studies Exact reproducibility in qualitative studies may not be possible. Focus on **transparency and traceability** rather than exact reproduction: * Document the experimental context clearly. * Support replication of procedures rather than exact results. * Explain how results may vary across different contexts or participant populations. * When raw qualitative data cannot be disclosed, share study protocols, coding schemes, coding rules, and intermediate analysis artifacts. * Document how anonymization may affect data interpretability and what limitations this introduces. ## Version control practices > **Key principle:** Version control systems preserve the history of decisions, scripts, data transformations, and workflow changes throughout the project. In practice, researchers work on local copies of their files and record changes in the repository whenever they want to create a permanent version or share progress. When multiple contributors edit the same files, the system detects overlapping changes and requires conflicts to be resolved before integrating contributions. ### What version control enables * Tracks the history of changes automatically, without manual file naming conventions or separate logs. * Stores only differences between versions rather than full copies, making the process efficient even for large projects. * Supports distributed collaboration by allowing multiple contributors to work simultaneously. * Provides an accurate record of what actually changed. ### What to version control Version control works best with **plain text files** such as source code, scripts, and documentation. Binary files (e.g., compiled executables, PDFs, Word documents) cannot be inspected in detail between versions, limiting the usefulness of version tracking for those formats. Guidelines: * **Raw data** typically does not require version control, as it should remain unchanged. * **Intermediate data and results** can be excluded if they can be regenerated from the original data and scripts. * **Small datasets and result files** may still be versioned when this supports collaboration and comparison across different workflow versions. > **Tip:** When code and data have different characteristics or distribution requirements, they may be maintained under separate version control or distribution strategies. ### Artifact versioning and persistent identifiers Platforms such as Zenodo and FigShare support two types of identifiers for versioned artifacts: * **Version-specific PIDs** point to a single, fixed version. Use these in the paper to reference the exact version used to produce the results. This ensures readers access the same materials. * **Concept PIDs (version-agnostic)** redirect to the latest version of the artifact. These are useful when the artifact may evolve during review, since the identifier remains valid without requiring the paper link to be updated. Each new version of the artifact should result in a new version-specific PID. See [Module 5 - Publishing](/gaps/jQBaAQpYuk4HtfLcsK7Y/m5.md) and [Module 6 - Maintenance](/gaps/jQBaAQpYuk4HtfLcsK7Y/m6.md) for guidance on versioning workflows and changelogs. ## Common pitfalls * Relying on undocumented manual steps that only work on the original development machine. * Using proprietary or non-reproducible tools without documenting alternatives. * Failing to track versions of data, scripts, and dependencies. * Uploading compressed archives that cannot be extracted on different operating systems. * Releasing artifacts with hardcoded machine-specific paths. * Overstating platform support for environments tested on only one machine. * Leaving expected warnings or error messages unexplained. * Not testing the artifact on a clean machine before sharing. * Releasing partial artifacts that do not support the main claims of the paper. ## Key takeaways * Reproducibility is multidimensional: accessibility, environment specification, versioning, execution fidelity, and legal openness all contribute. * **Automate workflows end-to-end.** Manual steps reduce reproducibility and make verification harder. * **Specify dependencies precisely** with pinned versions and machine-readable files. * **Prefer self-contained, containerized environments** over scripts that rely on external downloads. * **Validate the artifact in a clean environment** before releasing it, ideally with an external tester. * **Clean the artifact before release:** remove unused files, dead code, and materials not referenced in the paper. * **Document result variability honestly.** Justifiable deviations are acceptable; undocumented ones are not. * Include a **disclosure statement** describing what is included, excluded, and why. * Use **version control** throughout the project and assign version-specific PIDs for traceability. *** [Shared references used across modules.](/gaps/jQBaAQpYuk4HtfLcsK7Y/references.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://gaps.gitbook.io/gaps/jQBaAQpYuk4HtfLcsK7Y/m4.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.