> For the complete documentation index, see [llms.txt](https://gaps.gitbook.io/gaps/jQBaAQpYuk4HtfLcsK7Y/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://gaps.gitbook.io/gaps/jQBaAQpYuk4HtfLcsK7Y/m1.md).
# Module 1 - Foundations
← Previous module | [GAPS structure](/gaps/jQBaAQpYuk4HtfLcsK7Y/structure.md) | [Next module →](/gaps/jQBaAQpYuk4HtfLcsK7Y/m2.md)
***
## Module 1 - Foundations
Research in Software Engineering depends on more than well-written papers. For results to be trusted and built upon, other researchers must be able to access and reproduce the underlying materials. Research artifacts emerged to complement traditional, paper-based communication and make the research process more visible, verifiable, and reusable.
Before discussing artifacts in detail, it is important to understand the broader context in which they emerge. Open Science is the movement that motivates their creation, sharing, and reuse. Understanding these foundations is essential to grasp why artifacts play a central role in improving transparency and reproducibility, before engaging with the practical aspects covered in subsequent modules.
## Learning objectives
By the end of this module, you will be able to:
* Understand how Open Science motivates the creation, sharing, and reuse of research artifacts.
* Recognize the role of research artifacts in supporting transparent and reproducible research.
* Become familiar with different types of research artifacts and their roles in the research process.
* Recognize the characteristics of high-quality research artifacts.
* Apply the artifact life cycle to prepare, document, archive, and share artifacts.
## Open Science
The credibility and verifiability of empirical studies in Software Engineering depend strongly on transparency regarding complex artifacts, such as datasets, tools, and experimental setups. Historically, scientific communication was constrained by print-based publication models, which imposed strict limits on space and high dissemination costs. This often led researchers to present summarized methodological descriptions that mask the inherent complexity of the research process, such as intermediate decisions, discarded analyses, or changes in study design.
Today, networked technologies have removed these physical barriers, allowing scientific communication to move beyond the static paper and encompass the sharing of the entire research cycle. In this context, Open Science is not a set of universal prescriptions, but an invitation to be explicit and honest about research practices, a set of practices designed to increase the transparency of evidentiary reasoning and access to research across key stages of the research process.
The transition toward more transparent practices has been gradual and must account for contextual constraints, such as ethical considerations, participant privacy, or the sensitivity of proprietary software data. Ultimately, this movement seeks to improve the quality of scientific dialogue by addressing chronic issues like publication bias and replication failures through increased visibility of the entire research process.
In Software Engineering, these concerns contributed to the emergence of initiatives aimed at improving transparency, reproducibility, and long-term accessibility of research artifacts. One important example is the adoption of artifact evaluation processes by major conferences, encouraging authors to share datasets, tools, scripts, and other materials that support published findings. The Software Engineering community was among the early adopters of these practices. One of the earliest examples occurred at ESEC/FSE 2011, where research artifacts supporting published results were voluntarily submitted for peer review.
### What is Open Science?
Open Science is a set of practices that make the research process and its underlying reasoning more transparent. It makes scientific knowledge, data, methods, and processes openly available so that others can access, reuse, and build upon them. More specifically, Open Science:
* Promotes collaboration across researchers and society.
* Supports validation, reproducibility, and broader impact of research.
* Involves advocacy efforts to promote openness, influence policies, and engage stakeholders across the research ecosystem.
Scientific progress depends on research that is reliable, verifiable, and reusable. Transparency, credibility, and reproducibility are essential foundations for building robust knowledge, particularly in an evolving field such as Software Engineering.
### Open Science practices
Open Science encompasses a broad and evolving ecosystem of practices, principles, infrastructures, and policies. Since this ecosystem is extensive, this module focuses on the subset of practices most directly connected to research artifacts and reproducibility in empirical Software Engineering research.
#### Open Access
Scientific publications are made freely available on the public internet without financial, legal, or technical barriers, allowing anyone to read, download, copy, reuse, distribute, and link to full texts of publications for any lawful purpose.
* **Why it matters:** Open Access increases the visibility and dissemination of research, accelerates knowledge transfer, and reduces inequalities by allowing researchers worldwide to access scientific results regardless of institutional resources.
* **Relation to research artifacts:** Open Access ensures that the paper describing the artifact is available, enabling others to understand its context, usage, and contributions.
* **Examples in practice:**
* Publishing papers in open-access venues.
* Depositing preprints in repositories (e.g., arXiv).
* Providing author-accepted manuscripts in institutional repositories.
#### Open Data
Data produced during research are made available for access and reuse, typically through public repositories. It includes all data collected or generated during the research, as well as supporting materials. Openness can come in various forms and at different degrees, depending on ethical and legal constraints.
* **Why it matters:** Open Data enables validation of results, supports reproducibility, and allows secondary analyses. It increases the value of data (often costly to collect) and improves the quality of peer review by allowing inspection of underlying evidence.
* **Relation to research artifacts:** Open Data corresponds directly to data artifacts, which are central for reproducibility, validation, and reuse.
* **Examples in practice:**
* Publishing datasets in repositories (e.g., Zenodo, Figshare).
* Sharing replication packages with raw, processed, and derived data, including texts, interview transcripts, log data, and diaries.
* Providing metadata and data documentation.
#### Open Resources
Teaching, learning, and research materials in any medium, released under an open license that allows no-cost access, use, adaptation, and redistribution by others with no or limited restrictions.
* **Why it matters:** Open resources support learning, lower barriers to entry, and broaden access to scientific knowledge.
* **Relation to research artifacts:** Open Resources correspond to documentation and communication artifacts, which improve usability, understanding, and learning, supporting reuse and adaptation.
* **Examples in practice:**
* Sharing tutorials, lecture slides, and teaching materials.
* Publishing guidelines, checklists, and documentation.
* Providing reusable study materials.
#### Open Source (including open research software)
Open research software refers to software developed to be used in research (for analysis, simulation, or visualization) or developed as a research output, whose source code is made publicly available. It is shared in a user-friendly, human- and machine-readable, and modifiable format, under an open license that allows others to use, study, modify, and redistribute it.
* **Why it matters:** Open Source in research enables transparency in computational processes, supports reproducibility, and allows others to inspect, reuse, and extend research software.
* **Relation to research artifacts:** Open Source corresponds to implementation artifacts, enabling execution and reproduction of results.
* **Examples in practice:**
* Publishing code on GitHub with an open license.
* Sharing analysis scripts and pipelines.
* Providing executable implementations of algorithms.
#### Open Peer Review
Open Peer Review is an umbrella term for practices that aim to increase transparency in the peer review process. It may include open identities (authors and reviewers know each other), open reports (reviews are publicly available), and other forms of interaction and participation. There is yet no commonly accepted and clear definition nor an agreed schema.
* **Why it matters:** Open Peer Review increases transparency and accountability, improves the quality of feedback, and allows broader scrutiny of research, data, and methods.
* **Relation to research artifacts:** Open Peer Review increases scientific rigor through increased scrutiny of data and methods. It reinforces the need for sharing research artifacts to support verification and reproducibility.
**Examples in practice:**
* Publishing review reports alongside papers.
* Open discussion between authors and reviewers.
* Community-driven or post-publication review.
#### Open Methods
Research methods, protocols, and procedures are explicitly documented and shared, enabling others to understand and reproduce the research process.
* **Why it matters:** Open Methods improve transparency, reduce ambiguity in research design, and support replication by making methodological decisions explicit.
* **Relation to research artifacts:** Open Methods correspond to methodological artifacts, which are essential for replication and understanding how results were produced.
* **Examples in practice:**
* Sharing study protocols and experimental designs.
* Providing data collection instruments (e.g., surveys, interview scripts).
* Documenting analysis procedures and decisions.
#### FAIR principles
A set of guiding principles designed to ensure that digital research objects are **F**indable, **A**ccessible, **I**nteroperable, and **R**eusable.
* **Why it matters:** FAIR principles provide a structured way to improve the quality and reusability of research objects, ensuring they can be discovered, accessed, integrated, and reused by both humans and machines. Making data open does not guarantee reuse, so data must also follow FAIR principles.
* **Relation to research artifacts:** FAIR principles define the quality attributes of good artifacts, guiding how all types of artifacts should be structured, documented, and shared.
In practice, FAIR-aligned artifacts should be easy to find, access, interpret, connect with other resources, and reuse over time. Openness alone is not sufficient. FAIR principles complement Open Science by making research artifacts genuinely usable and reusable for both humans and machines.
**FINDABLE**
Artifacts are assigned persistent identifiers, described with rich metadata, and indexed in searchable resources.
| Principle | Description |
| -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **F1. Globally unique and persistent identifier (PID)** | Artifacts should have stable and unique identifiers that allow them to be reliably referenced, cited, and located over time. |
| **F2. Rich metadata** | Artifacts should include detailed metadata describing their content, purpose, authorship, structure, context, and usage. |
| **F3. Metadata include the PID of the described object** | Metadata should explicitly reference the identifier of the artifact they describe. |
| **F4. Registered or indexed in searchable resources** | Artifacts and their metadata should be deposited in repositories or platforms where they can be searched, discovered, and accessed by others. |
> **Note:** Persistent identifiers (PIDs) are typically resolvable through stable web links, allowing artifacts to remain locatable even if their storage location changes. DOI is the PID standard most widely adopted by archival repositories and supported across scholarly indexing infrastructures (e.g., Crossref, Scopus). Other PIDs also exist (e.g., Handle, ARK, SWHIDs).
**ACCESSIBLE**
Artifacts are retrievable via open, standardized protocols with defined access conditions, and metadata remain accessible even if the artifact itself is no longer available.
| Principle | Description |
| ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| **A1. Retrievable via PID using a standardized protocol** | Artifacts should be accessible through stable and standardized mechanisms (e.g., HTTPS, APIs) using their identifiers. |
| **A1.1. Open, free, and universally implementable protocols** | Access protocols should rely on open and widely supported technologies that do not require proprietary tools. |
| **A1.2. Authentication and authorization when necessary** | When artifacts cannot be fully public, access mechanisms should still support controlled and well-defined authorization procedures. |
| **A2. Metadata remain accessible even if the artifact is no longer available** | Even if the artifact becomes unavailable, its metadata should remain accessible so others can still understand its existence and context. |
**INTEROPERABLE**
Artifacts use shared vocabularies, formal languages, and include links to related data for integration.
| Principle | Description |
| ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| **I1. Formal, accessible, shared, and broadly applicable representation languages** | Artifacts should use standardized and broadly supported formats. |
| **I2. Vocabularies that follow FAIR principles** | Artifacts and metadata should adopt shared vocabularies or terminologies that are themselves well-documented and reusable. |
| **I3. Qualified references to related objects** | Artifacts should explicitly indicate how they relate to other datasets, scripts, papers, or research outputs. |
**REUSABLE**
Artifacts are well-described with relevant attributes, include clear usage licenses and provenance, and comply with community standards.
| Principle | Description |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| **R1. Richly described with accurate and relevant attributes** | Artifacts should provide sufficient detail about their content, context, assumptions, limitations, and intended use. |
| **R1.1. Clear and accessible usage license** | Artifacts should include explicit licensing information specifying how others may use, modify, and redistribute them. |
| **R1.2. Detailed provenance** | Artifacts should document their origin, history, processing steps, versions, and responsible contributors. |
| **R1.3. Domain-relevant community standards** | Artifacts should follow conventions, formats, and best practices commonly adopted within the relevant research community. |
### Why practice Open Science?
Practising Open Science may require additional effort, but it brings important benefits at multiple levels:
| Level | Benefits |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **For research** | - Supports transparency, validation, and reproducibility.
- Facilitates the reuse of results.
- Accelerates knowledge generation.
|
| **For society** | - Expands access to scientific knowledge regardless of geographic or economic constraints.
- Increases the return on publicly funded research.
|
| **For you** | - Increases visibility and potential impact of your work.
- Creates additional citable outputs (e.g., datasets, code).
- Fosters new collaborations and research opportunities.
|
To make Open Science practical, researchers rely on **research artifacts**, the concrete mechanism through which Open Science becomes actionable, translating abstract principles into tangible and reusable research outputs.
## Research artifacts
In the context of Open Science, sharing only the paper is not enough to fully communicate a study. To make research transparent, reproducible, and reusable, you also need to share the materials that support it.
### What is a research artifact?
A research artifact is any external material associated with a research report (e.g., a paper) that helps others understand, verify, reproduce, or reuse the study. These artifacts are made available via a link within the research report. More formally, a research artifact is a digital object generated as a result of the research itself or created by the authors to be used as part of the research, being essential for the associated paper.
### Why do artifacts matter?
* They make research **transparent and verifiable**, allowing others to understand and validate how results were produced.
* They support **reproducibility and replicability**, enabling studies to be re-executed under the same or similar conditions.
* They promote **reusability and cumulative science**, allowing future work to build upon, extend, and compare existing results.
## Types of research artifacts
Artifacts can be grouped into categories based on what they represent and how they contribute to the study. These categories are not strict or exhaustive, and a single artifact may fit into more than one category. Together, they reflect the full research lifecycle, from problem definition to execution, communication, and reuse.
| Artifact type | Definition | Role |
| --------------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------ |
| **1. Conceptual and scientific artifacts** | Artifacts that define, guide, or interpret the research. | Frame the research problem and contribute to scientific knowledge. |
| **2. Methodological artifacts** | Artifacts that describe how the research is conducted. | Enable understanding and replication of the research design. |
| **3. Data artifacts** | Artifacts related to the data used or produced. | Support empirical analysis and validation of results. |
| **4. Implementation artifacts** | Artifacts that operationalize the research. | Enable execution and reproduction of results. |
| **5. Documentation and communication artifacts** | Artifacts that explain how to understand and use the work. | Improve usability, accessibility, and learning. |
| **6. Reproducibility and infrastructure artifacts** | Artifacts that support execution across environments. | Ensure reproducibility and facilitate reuse. |
| **7. Research outputs as artifacts** | Artifacts traditionally seen as outputs that can also be reused. | Connect the artifact ecosystem to the published work. |
**Examples by category:**
| Artifact type | Examples |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **1. Conceptual and scientific** | - Motivational or challenge statements
- Hypotheses
- Baseline results
- New findings
- Negative results
- Future work directions
- Annotated or structured bibliographies
|
| **2. Methodological** | - Study instruments (e.g., surveys, interview scripts)
- Study protocols
- Sampling procedures
- Statistical tests and analysis rationale
- Checklists for study design
- Patterns (best practices)
- Anti-patterns (common pitfalls)
|
| **3. Data** | - Raw datasets
- Processed datasets
- Derived datasets
- Data documentation
|
| **4. Implementation** | - Analysis or visualization scripts
- Programs implementing algorithms
- Executable models
- Pipelines and workflows
|
| **5. Documentation and communication** | - README files
- Execution guides
- Tutorials and educational materials
- Commentary on scripts or analysis
- Informative visualizations
- Figures and tables used in the paper
|
| **6. Reproducibility and infrastructure** | - Configuration and dependency files
- Build scripts
- Containerization setups (e.g., Docker)
- Virtual machines or pre-configured environments
- Delivery tools for automated execution
|
| **7. Research outputs** | - The paper manuscript
- Supplementary materials
- Figures and tables
|
Research artifacts can range from simple non-executable materials, such as datasets, documentation, and study protocols, to fully executable artifacts, including software systems, automated pipelines, and reproducible environments. Even relatively small artifacts (such as the data used to generate figures or the scripts used in the analysis) can significantly improve transparency beyond what is possible in the paper alone. What matters most is that the artifact is relevant to the study and contributes meaningfully to understanding, verification, reproducibility, or reuse.
### Execution-oriented vs. transparency-oriented artifacts
Research artifacts can also be understood according to how they support verification, reuse, and reproducibility:
* **Execution-oriented artifacts** are primarily designed to support computational reproduction (running software, reproducing analyses, regenerating results, or recreating experimental workflows).
* Examples: executable scripts, pipelines, notebooks, containers, virtual machines, and automated workflows.
* **Transparency-oriented artifacts** primarily support inspection, interpretation, and understanding. These artifacts may not be directly executable, but they remain essential for explaining the study design, documenting decisions, clarifying procedures, and enabling critical assessment of the research.
* Examples: study protocols, interview guides, coding manuals, documentation, figures, and methodological descriptions.
Many artifacts combine both roles. For example, notebooks may simultaneously document and execute analyses, while datasets may support inspection alone or become executable when integrated into automated pipelines.
> **Important:** Non-executable artifacts are still valuable research artifacts. Transparency, documentation, and contextualization are fundamental components of reproducible and reusable research.
| Artifact example | Primary role |
| --------------------- | ------------------------------------------- |
| README | Transparency-oriented |
| Survey instrument | Transparency-oriented |
| Interview protocol | Transparency-oriented |
| Dataset | Transparency-oriented or execution-oriented |
| Analysis script | Execution-oriented |
| Jupyter notebook | Both |
| Docker container | Execution-oriented |
| Paper figures | Transparency-oriented |
| Reproduction pipeline | Execution-oriented |
## Artifact life cycle
Research artifacts should not be treated as a final step of the research process. Instead, they should evolve alongside the study, from its early stages to publication and beyond. Thinking in terms of a life cycle helps ensure that artifacts are not only created, but also properly prepared for sharing, reuse, and long-term preservation. Although presented as stages, this process is iterative: artifacts are refined over time, especially after feedback, reuse, or replication attempts.
| Stage | What to do | Why it matters |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| **1. Collect** | Gather all materials produced or used during the research (continuously, not only at the end). Includes data, code, scripts, protocols, and supporting resources. | Missing or incomplete materials are one of the main barriers to reproducibility. |
| **2. Document** | Clearly describe the artifact: purpose, scope, organization, authorship, relation to the paper, and execution instructions when applicable. | An undocumented artifact is effectively unusable. |
| **3. License** | Define how others can use, modify, and share the artifact. Use appropriate open licenses (e.g., MIT or Apache for code; CC BY for data). | Without a clear license, reuse is legally uncertain. |
| **4. Archive** | Store artifacts in reliable repositories that support long-term preservation, persistent identifiers, and versioning. | Personal websites and temporary links are not reliable for long-term access. |
| **5. Share** | Make artifacts visible and accessible to the research community. Link them directly in the paper. | Artifacts only contribute to science if others can find and access them. |
The practical guidance for each of these stages (how to prepare, document, ensure reproducibility, publish, and maintain artifacts) is covered in the subsequent modules.
## Key characteristics of good research artifacts
Good artifacts are not required to be perfect or exhaustive to be valuable. What matters most is that the artifact is relevant to the study and contributes meaningfully to understanding, verifying, or extending the research. These qualities can be understood through three complementary dimensions.
### Usability - *Can others understand and use it?*
Good artifacts should be understandable and usable without requiring extensive interpretation from the paper authors.
* **Well-documented.** Clearly describe purpose, structure, organization, dependencies, execution process, and expected outputs.
* README files, execution instructions, metadata, inline comments in scripts.
* **Self-contained.** Include all necessary components, or clearly specify external dependencies and how to obtain them.
* Datasets, scripts, configuration files, dependency versions, environment specifications.
* **Consistent.** Remain aligned with the paper and its claims, figures, tables, and execution procedures.
### Reproducibility - *Can others run and verify it?*
Good artifacts should allow others to independently execute the research workflow and verify how results were produced.
* **Executable.** Run with reasonable effort using the provided instructions and environment specifications.
* Avoid undocumented manual setup, hidden dependencies, or excessive configuration effort.
* **Automated.** Minimize manual intervention whenever possible. Automation reduces human error and improves consistency.
* Automated pipelines, preprocessing scripts, and automatic regeneration of figures and tables.
* **Verifiable.** Allow others to independently inspect and validate the research process and its outputs.
* Provide access to intermediate outputs, analytical procedures, and generated results.
### Reusability - *Can others build upon it?*
Well-prepared artifacts may support uses beyond the original study. This includes secondary analyses, benchmarking, teaching, comparative studies, or integration into future research workflows.
* **Legally compliant.** Include clear licensing and respect ethical, legal, and privacy constraints.
* This includes software licenses, data usage permissions, consent restrictions, third-party dependency compliance.
* **Preserved.** Store in reliable repositories with long-term access.
* Temporary or personal hosting solutions may become unavailable over time.
* **FAIR-aligned.** Follow FAIR principles so they can be found, accessed, integrated, and reused.
* This includes rich metadata, standardized formats, searchable repositories, explicit relationships between artifacts.
## Key takeaways
* Open Science needs artifacts.
* Artifacts support transparency and reproducibility.
* Good artifacts are **usable, reproducible, and reusable**.
* Artifact preparation is iterative.
* Artifacts should evolve throughout the project.
* Good artifacts should work independently of the paper.
***
[Shared references used across modules.](/gaps/jQBaAQpYuk4HtfLcsK7Y/references.md)
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://gaps.gitbook.io/gaps/jQBaAQpYuk4HtfLcsK7Y/m1.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.