Frequently Asked Questions

OrangeFS is an open source, multi-server parallel file system for production and research clusters. It is designed to scale to large numbers of clients and servers, with a modular architecture so new networks, storage backends, and algorithms can be added. Sites use it for shared POSIX trees across many nodes: simulation checkpoints, research lab storage, and AI training or fine-tuning I/O.

Servers and admin tools run in user space. The usual Linux client path uses an upstream kernel client (mainline since Linux 4.6), so mounts do not require shipping an out-of-tree client module with every node image. A high-performance user-space interface is also available when you want to bypass the kernel path.

OrangeFS grew out of the PVFS lineage that began at Clemson University in the 1990s (originally for parallel I/O research, later production HPC). PVFS2 redesigned the system around object servers, distributed metadata, MPI-oriented views, and pluggable networks. A Clemson-led branch focused on security, redundancy, and a broader range of systems; in 2010 that line became OrangeFS, the mainline project going forward.

In 2016 the OrangeFS client was merged into the Linux kernel (4.6). Current 2.x releases continue maintenance and features while longer-term resiliency work targets OrangeFS v3. See the Wikipedia article and the projects page for more.

PVFS (Parallel Virtual File System) is the research and production parallel file system lineage that OrangeFS continues. For practical purposes, OrangeFS is the maintained successor: same architectural family, current releases, and community under the OrangeFS name. Older docs and tools may still say PVFS or use pvfs2-* binary names; those map to OrangeFS components.

Omnibond provides commercial support, professional services, and cloud marketplace options for OrangeFS while the software remains open source. Community development continues on GitHub; Omnibond contributes engineering and supports production deployments. Enterprise and cloud paths live on orangefs.com; the open source project site is orangefs.org.

orangefs.org is the open source project: releases, FAQ, community, roadmap, and links to docs and GitHub.

orangefs.com is support and services: enterprise help, deployment assistance, and cloud marketplace options when self-supported open source is not enough.

OrangeFS is released under the LGPL. Some recent portions are also available under LGPL and Apache 2. Confirm license files in the release tarball or GitHub repository for the version you ship.

At a high level:

• Servers (historically pvfs2-server): store data and/or metadata in user space on local disks.
• Linux kernel client: upstream VFS module for mount -t orangefs (and related tooling).
• User-space client / admin tools: configuration, ping/statfs-style utilities, and optional high-performance user interface under src/client/usrint.
• BMI: network abstraction used between clients and servers (TCP, InfiniBand, and other modules depending on build).

Full architecture and install steps are in the OrangeFS documentation.

Release versions follow a major.minor.patch style (for example 2.10.1). The current open source tarball and Windows client are published on the project GitHub releases page. Older packages and docs may still mention PVFS2 versioning; treat OrangeFS 2.x as the supported 2.x product line.

Check GitHub Releases for the latest tag, ChangeLog, and download artifacts.

v3 is the main longer-term design track: a new data model aimed at stronger file and metadata resiliency, more flexible server membership, and background maintenance. Most active development is described on the projects page. Production sites today typically run maintained 2.x releases while v3 work continues.

No specialized appliance is required. Servers use disks local to each node (or local RAID/LVM volumes). Networking uses standard cluster fabrics. TCP is the common default; InfiniBand and other BMI modules can be built when your site needs them. A SAN is not required, though shared storage can still appear in high-availability designs.

Most of OrangeFS is portable user-space C. Production deployments today are overwhelmingly on 64-bit Linux (x86_64 and other mainstream Linux architectures). Historical testing also covered other platforms; for a new site, assume Linux servers plus the clients described below, and verify your architecture against current release notes if you are off the common path.

Yes. On-wire encoding uses a fixed, architecture-independent layout so clients and servers on different CPU types can interoperate when builds and versions are compatible.

User-space servers and tools are intended to run on current major GNU/Linux distributions. The in-tree OrangeFS client has been in mainline Linux since 4.6. Use a distribution kernel that includes OrangeFS support (or build a mainline-based kernel with the client enabled). You do not need an out-of-tree client package on every node image the way some parallel file systems require.

Attribute caching improvements in Linux 4.9+ reduce metadata traffic for common workloads. Prefer a maintained distribution kernel rather than random pre-release (-rc) kernels unless you are actively testing with the community.

No. Prefer the upstream client in your distribution or mainline kernel. Distribution-specific packaging may still matter for userspace tools and server packages; for client access, the integrated module is the supported path.

Yes. A Windows client installer is published with recent releases on GitHub (for example the 64-bit client next to the 2.10.1 source tarball). See the release assets on GitHub Releases.

macOS can use FUSE-based access via the OrangeFS FUSE module (for example with macFUSE). WebDAV access is also available through the dav_orangefs module built from the distribution. Neither is as common as the Linux kernel client; treat them as optional access paths and test against your release.

Linux containers can mount OrangeFS when the host provides the kernel client and the mount is available to the container (bind mount or privileged mount patterns, depending on your runtime). There is no separate “OrangeFS operator” required by the project itself. For production cloud marketplace images and supported cloud layouts, see orangefs.com.

Start with the OrangeFS documentation for single-server and multi-server installs, then download a current release from GitHub. Prefer distro packages when your distribution provides maintained OrangeFS packages and they match the version you intend to run.

Confirm checksums and release notes before deploying. For enterprise installers and cloud AMIs, use orangefs.com.

Usually you present one logical volume to the OrangeFS server: hardware RAID, Linux MD, or LVM (or a combination), then put the server storage space on that filesystem. That keeps a single server process simple while still using many spindles or SSDs.

A second approach is multiple OrangeFS server instances on the same host, each with its own storage path and config, when disks need to be exposed as separate servers. Separate I/O paths help when the hardware can feed them concurrently.

Yes. Treat each instance like a separate server: its own Alias entry, server-specific configuration, and storage space, as described in the documentation.

Yes. Any OrangeFS server can hold metadata, data, or both. Allocate unique metadata handle ranges for each metadata server; clients use the configuration to locate metadata. Distributed directory entries (GIGA+-style) help very large directories; see completed work on the projects page.

Multiple metadata servers distribute load and capacity; they do not by themselves fully replicate metadata for arbitrary failure. Back up metadata regularly, and review fault-tolerance options (local RAID, HA recipes, immutable-file redundancy features). Stronger resiliency is a major theme of v3 work.

It depends on the workload.

• Metadata servers: more help when many clients create or look up different files (small-file heavy trees).
• Data servers: more increase aggregate bandwidth for large sequential or strided I/O (checkpoints, large datasets), provided clients issue large enough requests to stripe effectively.

Dedicated I/O nodes usually outperform “server on every compute node” for peak bandwidth, though co-located servers are supported when the cluster is small or budget-constrained.

Yes. OrangeFS servers can listen on multiple interfaces so different client networks or dual-plane fabrics can reach the same service.

A typical modern mount looks like:

mount -t orangefs tcp://server0:3334/orangefs /mnt/orangefs

For automount, escape the colon in the resource string so the automounter does not mangle the URL, for example:

orangefs -fstype=orangefs tcp://server0\:3334/orangefs

Exact package and module names can vary by distribution; if orangefs is not accepted, check whether your distro still documents a pvfs2 type alias for the same client.

Yes. Each file system needs unique Name and ID values in its configuration. Do not clone an unedited fs.conf for a second file system; adjust identifiers before creating storage spaces.

OrangeFS can often be installed over a historical PVFS2 deployment, with metadata updates applied at startup. Always back up metadata and data first, read the ChangeLog for your target version, and test on a non-production copy. Some backend changes (for example BDB to LMDB) have no in-place conversion and require a new filesystem or explicit migration.

LMDB is the default metadata option in modern OrangeFS and generally performs better than the older Berkeley DB path. There is no in-place upgrade from BDB to LMDB; plan a new filesystem (or a data movement strategy) when changing backends. See completed project notes on the projects page.

Primary documentation: docs.orangefs.com. Releases, source, and ChangeLogs: github.com/waltligon/orangefs. Project overview and roadmap: this site (home, FAQ, community, projects).

1. Check the docs and this FAQ for known issues (mount syntax, performance expectations, limits).

2. Search GitHub Discussions and issues for similar reports.

3. Open a Discussion or issue with enough detail to reproduce (see next question). For production support contracts, contact Omnibond via orangefs.com.

Use GitHub Discussions for questions and design talk, and GitHub issues for actionable defects. Include:

• OrangeFS version (and whether servers/clients match)
• Linux distribution and kernel version (for VFS mounts)
• MPI / MPI-IO stack if relevant
• Rough hardware sketch: node counts, network, storage layout
• Exact commands, errors, and useful log excerpts

Prefer the latest stable release when you can reproduce on it. Keep discussion in public GitHub threads so others can learn from the answer.

Typical causes:

• Typo in the mount command or resource string
• Kernel client module not loaded or not present in this kernel
• Userspace helpers not running or unable to reach servers (verify with the project’s ping/stat tools from the docs)
• Second file system reusing the same Name or ID as one already mounted (see 3.9)

Start from a documented mount example for your release, confirm servers are up, then inspect kernel logs (dmesg) for client-side errors.

OrangeFS links against libraries chosen at configure time (database backend, optional network stacks). If those libraries are not on the default loader path, either install them system-wide and run ldconfig, or set LD_LIBRARY_PATH consistently for service accounts. Prefer packaging that encodes the correct RPATH/dependencies for production.

Daily cron jobs that walk the entire filesystem (locate/updatedb/mlocate/plocate) often crawl network mounts and can hammer metadata. Exclude the OrangeFS mount point or filesystem type from those jobs on every client, the same way many sites exclude NFS.

Extended attributes are name/value pairs on files and directories beyond classic mode bits and timestamps. On Linux you typically use setfattr / getfattr. OrangeFS also uses some internal attribute names; avoid colliding with reserved keys used by the system (see docs for the current list). Directory “hints” for distribution and striping are set via user attributes (see Performance).

OrangeFS can store POSIX ACLs as extended attributes. Mount with ACL support enabled, for example:

mount -t orangefs -o acl tcp://server0:3334/orangefs /mnt/orangefs

Use setfacl / getfacl as on other Linux filesystems. Capability-based access control for distributed identity is also part of modern OrangeFS security work; see docs and the projects page.

Yes in principle: if the web server host can mount OrangeFS (or access it via an allowed interface), it can serve files like any other local mount. Watch consistency and performance expectations: OrangeFS is optimized for parallel scientific and data-intensive I/O, not necessarily small random web-object workloads. Test caching and concurrency under your real traffic pattern.

Storage format versions change across major upgrades. If tools report a metadata format mismatch, stop and follow the migration notes for your from/to versions rather than forcing an old server on new storage (or the reverse). Keep backups of the server storage directories before running migration utilities.

Classic serial tools issue many tiny reads and writes. Without aggressive client-side caching (OrangeFS historically prioritizes scalable consistency for parallel jobs), each small request pays network and server costs. Use larger block sizes, parallel benchmarks (for example IOR-style workloads), and application-like access patterns when evaluating the system.

Optional caching semantics are an active research/product topic; do not assume local-POSIX cache behavior unless you have configured and accepted those semantics.

Check request size first: small, synchronous I/O from every rank will underutilize the stripe. Review server config knobs that trade durability for throughput (sync behavior on metadata and data), and whether attribute caching is helping metadata-heavy phases. For single-node serial apps, a local or NFS volume may simply be a better fit.

On a single client talking to a single server with small I/O, NFS often wins because of client caching and different consistency choices. OrangeFS is aimed at many clients and large parallel transfers. If your real workload is serial and small, OrangeFS may not be the right tool. If the workload is multi-node checkpoints or wide dataset reads, retest with parallel I/O and larger transfers.

Yes. Each server stores objects on a local filesystem; fsync and allocator behavior differ by type and mount options. Sites often use a reliable Linux filesystem tuned for their disks (historically ext4/xfs-class setups are common). Measure with your disk subsystem rather than assuming defaults are optimal.

Yes. Directory hints via extended attributes can set distribution name, distribution parameters (such as strip size), and number of datafiles for newly created files (and inherited by new subdirectories). Example pattern:

setfattr -n user.pvfs2.dist_name -v basic_dist /mnt/orangefs/dir
setfattr -n user.pvfs2.dist_params -v strip_size:65536 /mnt/orangefs/dir
setfattr -n user.pvfs2.num_dfiles -v 8 /mnt/orangefs/dir

Exact attribute names and supported distributions are defined in your installed headers/docs; verify against your release.

When multiple interconnects are enabled, the stack may poll more than one path. For peak performance on a single fabric, build and run with the interconnect you actually use. Multi-network builds remain useful when you truly need flexibility.

• Are servers dedicated I/O nodes or co-located with heavy compute?
• Have you measured raw network and disk independently of OrangeFS?
• Are TCP (or IB) buffer and fabric settings sane under concurrent load?
• Is the application doing small serial I/O or wide parallel I/O?
• Do all clients and servers run matching software versions?

Share those answers when you open a GitHub Discussion; it shortens the loop substantially.

Yes, with configuration and often additional hardware or operational practice. Client failures are handled lightly by design; disk, network, and server failure require RAID, multipath networks, HA recipes, and/or redundancy features depending on your goals. v3 work targets stronger resiliency in the data model itself.

Yes. There is no global lock service and little shared client-side state that must be recovered for the cluster to continue. A lost client does not require fencing peers the way a tightly locked shared filesystem might.

At the server, use local redundancy (hardware RAID, MD, or equivalent) under the storage directory so a single disk failure does not take out that server’s objects. Application-level or feature-level redundancy (for example redundancy for immutable files) can add another layer; see docs and projects notes.

If the fabric provides redundant paths and hosts are multihomed appropriately, clients can continue when a single link fails. OrangeFS does not replace a well-designed network; it rides on top of it.

Server loss is the hard case. Documented high-availability approaches use shared storage and heartbeat-style failover (see HA material in the distribution docs). Software-only replication with low overhead is an ongoing design concern (including v3). Do not assume that a multi-server stripe alone survives losing an arbitrary server without an HA or redundancy plan.

Use an MPI stack whose ROMIO (or equivalent) build supports the OrangeFS/PVFS2 ADIO driver, and point jobs at a mounted OrangeFS path or the configured filesystem. Build notes vary by MPI distribution; see the OrangeFS docs and your MPI vendor’s parallel I/O guide. For maximum performance, also evaluate the user-space interface in src/client/usrint.

You can see objects under each server’s storage directory, but you should not hand-edit or rearrange them as if they were a normal POSIX tree. The layout is an internal representation. Use client mounts, supported tools, and documented admin procedures. Unsupported offline edits are a common way to strand data.

OrangeFS includes a multi-layer direct client (POSIX-like, stdio, MPI-IO paths) that can bypass the kernel for latency-sensitive or bandwidth-heavy applications. Sources live under src/client/usrint. Use it when the VFS path is the bottleneck and your application can link against the provided libraries.

Default storage (Trove DBPF-style layouts) keeps each server’s data and metadata under a server storage directory tree of ordinary files. Back up those trees with your usual site tools (snapshots, rsync, enterprise backup), consistently across servers, and test restores. Coordinate with unmounts or idle periods when you need a clean point-in-time copy.

You can add servers and reorder handle ranges, typically with all clients unmounted and servers stopped, config updated and distributed, storage created on new nodes, then services restarted. New servers receive new objects by policy; existing objects do not automatically rebalance without migration tooling.

General-purpose rebalancing/migration tooling has historically been limited. Plan capacity so you are not forced into emergency reshuffles, and watch the projects and release notes for staging helpers and v3 membership work. For research data movement into and out of OrangeFS, see data staging notes on the projects page.

Free space is reported conservatively: often based on the minimum free space on any data server times the number of data servers, so one full or unevenly filled server pulls the total down. Use OrangeFS statfs-style utilities to inspect per-server capacity and rebalance load outside the filesystem if needed.

Offsets use wide integer types, so software limits are very high. Practical limits come from local filesystems under each server, number of servers, and operational constraints. Estimate capacity as (usable space per server) times (data servers), minus headroom for metadata and imbalance.

Yes. The kernel client supports an interruptible mount option (commonly intr) so waits on a stuck filesystem can be interrupted. Example:

mount -t orangefs -o intr tcp://hosta:3334/orangefs /mnt/orangefs

Confirm option names for your kernel/docs revision.

Hard links are not implemented; symbolic links are. This is a long-standing design choice, not a temporary bug.

Private, read-only mmap is supported. Shared writable mmap is not, because supporting it would force heavyweight shared-state machinery that fights the parallel, low-lock design. Applications that require shared mmap should use another storage path.

Not in the sophisticated “always fullest-first” sense many admins want. Default placement does not fully optimize for free-space balance. Keep servers evenly provisioned and monitor per-server fill. Smarter placement remains a reasonable enhancement area for contributors.

No global lock service is built into the core design. Avoiding distributed locks keeps failure handling simpler for parallel I/O. Applications should use MPI or application-level coordination when mutual exclusion is required. Optional locking subsystems have been discussed historically but are not a required part of the core.

Report bugs with reproducers, improve docs, fix issues, or propose features. Start on GitHub Discussions, browse open issues, and read the community and projects pages. Patches and small documentation fixes are welcome; you do not need a long project history to help.

See the projects page for in-progress work (v3, data staging, Grafana stats), ideas (searchable metadata, small I/O), and completed milestones (mainline client improvements, LMDB metadata, capability-based access control, distributed directories, and more).

Yes when jobs need a shared POSIX namespace across many nodes and aggregate bandwidth or parallel checkpoint I/O matters: multi-node training and fine-tuning, MPI-IO simulation output, and shared lab or instrument data. It is less ideal as a drop-in home-directory NFS replacement for tiny serial I/O. Match the filesystem to the access pattern, and use the support site when you need a packaged enterprise or cloud deployment.

BMI is the network abstraction OrangeFS uses between clients and servers. Each BMI module (TCP, InfiniBand, and others depending on the build) defines its own maximum message size; higher layers query the module so buffers stay within limits. Sends matched with too-small receive buffers fail rather than silently truncating. See the BMI sources and docs if you are adding or tuning a network module.