In Linux Find TruthBlogging about Linux & Other ThingsIn Linux Find Truthhttps://teletype.in/atom/datapioneer2026-04-11T14:44:07.545Zdatapioneer:BJFUzIIkUZettabyte File System Explained2019-12-29T19:54:49.370Z2020-01-03T17:18:16.191ZIn this article, I will strive to answer many questions that have been asked about ZFS, such as what is it, why should I use it, what can I do with it, and the like? Let's begin:
<p>In this article, I will strive to answer many questions that have been asked about ZFS, such as what is it, why should I use it, what can I do with it, and the like? Let's begin:</p>
<h3>What are some of the attributes of ZFS?</h3>
<ul>
<li>ZFS is a fully-featured filesystem</li>
<li>Does data integrity checking</li>
<li>Uses snapshots</li>
<li>Created by Sun Microsystems, forked by Oracle</li>
<li>Oracle version is less full featured</li>
<li>OpenZFS - open source version of ZFS</li>
<li>Feeds into FreeBSD, illumos, ZFSonLinux, Canonical</li>
</ul>
<h3>What makes ZFS special?</h3>
<ul>
<li>Leverages well-understood standard userland tools into the filesystem</li>
<ul>
<li>Checksums everything</li>
<li>Metadata abounds</li>
<li>Uses Compression</li>
<li>diff(1)</li>
<li>Copy-on-Write (COW)</li>
</ul>
</ul>
<h3>What is Copy-on-Write?</h3>
<ul>
<li>ZFS never changes a written disk sector</li>
<li>A sector changes? Allocate a new sector. Write data to it</li>
<li>Data on disk is always coherent</li>
<li>Power loss half-way through a write? Old data is still there untouched. Version control at the disk level</li>
<li>Interesting side-effect. You can effectively free snapshots</li>
</ul>
<h3>ZFS Assumptions?</h3>
<ul>
<li>ZFS is not your typical EXT/UFS filesystem</li>
<li>Traditional assumptions about filesystems will come back to haunt you</li>
<li>Non-ZFS tools like <strong><em>dump</em></strong> will <em>appear</em> to work, but not really</li>
</ul>
<h3>ZFS Hardware?</h3>
<ul>
<li>RAID Controllers -- Absolutely NOT!</li>
<ul>
<li>ZFS expects raw disk access</li>
<li>RAID controller in JBOD or single-disk RAID0?</li>
<li>RAM -- ECC?</li>
<li>Disk redundancy</li>
</ul>
</ul>
<h3>ZFS Terminology</h3>
<ul>
<li>VDEV or Virtual Device - a group of storage providers</li>
<li>Pool - a group of identical VDEVs</li>
<li>Dataset - a named chunk of data on a pool</li>
<li>You can arrange data in a pool anyway that you desire</li>
<li><strong>-f</strong> switch is very important (be careful how you use it)</li>
</ul>
<h3>Virtual Devices (VDEVs) and Pools</h3>
<ul>
<li>Basic unit of storage in ZFS</li>
<li>All ZFS redundancy occurs at the virtual device level</li>
<li>Can be built out of any storage provider</li>
<li>Most common providers: disk or GPT partition</li>
<ul>
<li>Could be FreeBSD crypto device</li>
<li>Low-Level Virtual Machine (LLVM) RAID</li>
</ul>
<li>A Pool contains only one type of VDEV</li>
<li>"X VDEV" and "X Pool" get used interchangeably</li>
<li>VDEVs are added to Pools</li>
<li>Typically providers are not added to VDEVs but to Pools</li>
</ul>
<h3>Stripe VDEV/Pool</h3>
<ul>
<li>Each disk is its own VDEV</li>
<li>Data is striped across all VDEVs in the Pool</li>
<li>Can add striped VDEVs to grow Pools</li>
<li>No redundancy. Absolutely none. Nada!</li>
<li>No self-healing</li>
<li>Set <em>copies=2</em> to get self-healing; Must be done when added</li>
</ul>
<h3>Mirror VDEV/Pool</h3>
<ul>
<li>Each VDEV contains multiple disks that replicate the data of all other disks in the VDEV</li>
<li>A Pool with multiple VDEVS is synonymous to RAID-10 (Stripe over Mirrors)</li>
<li>Can add more mirror VDEVs to grow Pool</li>
</ul>
<h3>RAIDZ VDEV/Pool</h3>
<ul>
<li>Each VDEV contains multiple disks</li>
<li>Data integrity maintained via parity (such as RAID-5, etc.)</li>
<li>Lose a disk - No data loss</li>
<li>Can self-heal via redundant checksums</li>
<li>RAIDZ Pool can have multiple identical VDEVs</li>
<li>Cannot expand the size of a RAIDZ VDEV by adding more disks</li>
</ul>
<h3>RAIDZ Types</h3>
<ul>
<li>RAID-Z1</li>
<ul>
<li>3+ Disks</li>
<li>Can lose 1 disk/VDEV</li>
</ul>
<li>RAID-Z2</li>
<ul>
<li>4 + Disks</li>
<li>Can lose 2 disks/VDEV</li>
</ul>
<li>RAID-Z3</li>
<ul>
<li>5+ Disks</li>
<li>Can lose 3 disks/VDEV</li>
</ul>
<li>Disk size far exceeds disk access speed</li>
</ul>
<h3>Number of Disks and Pools?</h3>
<ul>
<li>No more than 9 - 12 Disks per VDEV</li>
<li>Pool size is your choice</li>
<li>Avoid putting everything in one massive Pool</li>
<li>Best practice is to put OS in one mirrored Pool, and data in a separate Pool</li>
</ul>
<h3>RAIDZ vs. Traditional RAID</h3>
<ul>
<li>ZFS combines filesystem and Volume Manager - faster recovery</li>
<li>Write hole</li>
<li>Copy-on-Write -- never modify a block, only write new blocks</li>
</ul>
<h3>Create Striped Pools</h3>
<ul>
<li>Each VDEV is a single disk</li>
<li>No special label for VDEV of striped disk</li>
<li># zpool create trinity gpt/zfs0 gpt/zfs1 \ gpt/zfs2 gpt/zfs3 gpt/zfs4</li>
</ul>
<h3>Viewing Stripe/Mirror/RAIDZ Pool Results</h3>
<ul>
<li>Use # zpool status</li>
</ul>
<h3>Multi-VDEV RAIDZ</h3>
<ul>
<li>Stripes are inherently multi-VDEV</li>
<li>There's no traditional RAID equivalent</li>
<li>Use <em>type</em> keyword multiple times</li>
<ul>
<li># zpool create trinity <strong>raidz1</strong> gpt/zfs0 gpt/zfs1 gpt/zfs2 <strong>raidz1</strong> gpt/zfs3 gpt/zfs4 gpt/zfs5</li>
</ul>
</ul>
<h3>Malformed Pool Example</h3>
<ul>
<li># zpool create trinity <strong>raidz1</strong> gpt/zfs0 gpt/zfs1 gpt/zfs2 <strong>mirror</strong> gpt/zfs3 gpt/zfs4 gpt/zfs5</li>
<ul>
<li>receives an "invalid vdev specification" message</li>
</ul>
<li>Don't use <strong>-f</strong> here as ZFS will let you when you shouldn't</li>
<li>Attempting to add a Striped Mirror to a RAIDz -- No go!</li>
</ul>
<h3>Reusing Providers</h3>
<ul>
<li># zpool create db gpt/zfs1 gpt/zfs2 gpt/zfs3 gpt/zfs4</li>
<li>/dev/gpt/zfs3 is a part of an exported pool 'db' , so</li>
<li>the use of <strong>-f</strong> here is appropriate and essential</li>
</ul>
<h3>Pool Integrity</h3>
<ul>
<li>ZFS is self-healing at the Pool and VDEV level</li>
<li>Parity allows data to be rebuilt</li>
<li>Every block is hashed; hash is stored in the parent</li>
<li>Data integrity is checked as the data is accessed on the disk</li>
<li>A <strong>Scrub</strong> is essentially checking the live filesystem without off-line'ing</li>
<li>If you don't have VDEV redundancy, use dataset <em>copies</em> property</li>
</ul>
<h3>Scrub vs fsck</h3>
<ul>
<li>ZFS has no offline integrity checker</li>
<li>ZFS scrub does everything that fsck does, and more</li>
<li>You can offline your Pool to scrub, by why would you?</li>
<li>Scrub isn't perfect, but it's better than fsck</li>
</ul>
<h3>Pool Properties</h3>
<ul>
<li>Properties are tunables</li>
<li>Both Pools and Datasets have properties</li>
<li>Commands: <em>zpool set</em>, and <em>zpool get</em></li>
<li>Some are read-only</li>
<ul>
<li># zpool get all | less</li>
</ul>
</ul>
<h3>Changing Pool Properties</h3>
<ul>
<li># zpool set comment="Main OS Files" zroot</li>
<li># zpool set copies=2 zroot</li>
</ul>
<h3>Pool History</h3>
<ul>
<li># zpool history zroot</li>
</ul>
<h3>ZPool Feature Flags</h3>
<ul>
<li>ZFS had version numbers</li>
<li>Then, Oracle assimilated Sun</li>
<li>Feature flags are at version 5000</li>
<li>Feature flags versus OS</li>
<li># zpool get all trinity | grep feature</li>
</ul>
<h3>Datasets</h3>
<ul>
<li>A named chunk of data</li>
<li>Filesystems</li>
<li>Volume</li>
<li>Snapshot</li>
<li>Clone</li>
<li>Bookmark</li>
<li>Properties and features work on a per-dataset basis</li>
<li># zfs list -r zroot/ROOT</li>
</ul>
<h3>Creating Datasets</h3>
<ul>
<li>$ zfs create zroot/var/mysql</li>
<li>$ zfs create -V 4G zroot/vmware</li>
</ul>
<h3>Destroying Datasets</h3>
<ul>
<li># zfs destroy zroot/var/old-mysql</li>
<li><strong>-v</strong> -- verbose mode</li>
<li><strong>-n</strong> -- no-op flag</li>
</ul>
<h3>Parent-Child Relationships</h3>
<ul>
<li>Datasets inherit their parent's properties</li>
<li>If you change it locally, but want to have it use the parent's inherited property, use </li>
</ul>
<p> <em>zfs inherit</em></p>
<ul>
<li>Renaming a Dataset changes its inheritance</li>
</ul>
<h3>Pool Repair & Maintenance</h3>
<ul>
<li>Resilvering</li>
<li>Rebuild from parity</li>
<li>Uses VDEV redundancy data</li>
<li>No redundancy? No resilvering</li>
<li>Throttled by Disk I/O</li>
<li>Happens automatically when disk is replaced</li>
<li>Can add VDEVs to Pools, not disks to VDEV</li>
<li>Be cautious of slightly smaller disks (check sector size as they can vary from disk to disk of equal capacity)</li>
</ul>
<h3>Add VDEV to Pool</h3>
<ul>
<li>New VDEVs must be identical to existing VDEVs in the Pool</li>
<ul>
<li># zpool add scratch gpt/zfs99</li>
<li># zpool add db mirror gpt/zfs6 gpt/zfs7</li>
<li># zpool add trinity raidz1 gpt/zfs3 gpt/zfs4 gpt/zfs5</li>
</ul>
</ul>
<h3>Hardware States in ZFS</h3>
<ul>
<li>ONLINE -- operating normally</li>
<li>DEGRADED -- at least one storage provider has failed</li>
<li>FAULTED -- generated too many errors</li>
<li>UNAVAIL -- cannot open storage provider</li>
<li>OFFLINE -- storage provider has been shut down</li>
<li>REMOVED -- hardware detection of unplugged device</li>
<li>Errors percolate up through the ZFS stack</li>
<li>Hardware RAID hides errors - ZFS does not!</li>
</ul>
<h3>Log and Cache Devices</h3>
<ul>
<li>Read Cache -- L2ARC (Level 2 Adaptive Replacement Cache)</li>
<li>Synchronous Write Log -- ZIL, SLOG (ZFS Intent Log, Separate Log Device)</li>
<li>Where is the bottleneck?</li>
<li>Log/Cache Hardware</li>
</ul>
<h3>Filesystem Compression</h3>
<ul>
<li>Compression exchanges CPU time for disk I/O</li>
<li>Disk I/O is very limited</li>
<li>CPU time is plentiful</li>
<li>LZ4 by default</li>
<li>Enable compression before writing any data</li>
<li># set compress=lz4 zroot</li>
<li>Typically gzip-9 is better than lz4</li>
<li>No more userland log compression</li>
</ul>
<h3>Memory Cache Compression</h3>
<ul>
<li>Advanced Replication Cache is ZFS' Buffer Cache</li>
<li>ARC compression exchanges CPU time for memory</li>
<li>Memory can be somewhat limited</li>
<li>CPU time is plentiful</li>
<li>ZFS ARC auto compresses what can be compressed</li>
</ul>
<h3>Deduplication (Dedup)</h3>
<ul>
<li>ZFS deduplication isn't good as you would imagine it is</li>
<li>Only duplicates identical filesystem blocks</li>
<li>Most data is not ZFS deduplicable</li>
<li>1TB of dedup'd data = 5G RAM for the dedup process</li>
<li>Dedup RAM X 4 = System RAM</li>
<li>Effectiveness: run zdb -S zroot, check dedup column</li>
<li>Cost-effective ZFS dedup just doesn't exist</li>
</ul>
<h3>Snapshots</h3>
<ul>
<li># zfs snapshot trinity/home@<today's date></li>
<li># zfs list -t snapshot</li>
<li># zfs -r zroot@<today's date></li>
<li>Access snapshots in hidden .zfs directory (especially when using ZFSonLinux)</li>
<li># zfs destroy trinity/home@<today's date></li>
<li>Use -vn in destroy operations</li>
</ul>
<h3>Snapshot Disk Use</h3>
<ul>
<li>Delete file from live filesystem</li>
<li>Blocks in a snapshot remain in use</li>
<li>Blocks are freed only when no snapshot uses them</li>
</ul>
<h3>Roll Back</h3>
<ul>
<li>Can rollback filesystem to the most recent snapshot</li>
<li># zfs rollback zroot/ROOT/@<before upgrade></li>
<li>Newer data is destroyed</li>
</ul>
<h3>Clones</h3>
<ul>
<li>A read-write copy of a snapshot</li>
<li># zfs clone zroot/var/mysql@<today> \ zroot/var/mysql-test</li>
<li>Run a test, then discard afterward</li>
</ul>
<h3>Boot Environments</h3>
<ul>
<li>Built on clones and snapshots</li>
<li>Snapshot root filesystem dataset before an upgrade</li>
<li>If Upgrade goes awry. Roll back!</li>
<li>FreeBSD: sysutils/beadm</li>
</ul>
<h3>ZFS Send/Receive</h3>
<ul>
<li>Move whole filesystems to another host</li>
<li>Blows rsync out of the water</li>
<li>Resumable</li>
</ul>
<p><a href="https://inlinuxveritas.com/Sk68PBb1U?cda=" target="_blank">Let's install ZFS in Linux and start using it</a></p>
datapioneer:HkV-xNH18Designating Hot Spares in Your ZFS Storage Pool2019-12-28T21:24:52.001Z2019-12-28T23:19:45.794Z<img src="https://teletype.in/files/ed/1b/ed1bcab0-dc11-4c42-8920-a8f05e14ee7d.png">There is a feature built into ZFS called the "hotspares" feature which allows a sysadmin to identify those drives available as spares which can be swapped out in the event of a drive failure in a storage pool. If an appropriate flag is set in the feature, the "hot spare" drive can even be swapped automatically to replace the failed drive. Or, alternatively, a spare drive can be swapped manually if the sysadmin detects a failing drive that is reported as irreparable.
<p>There is a feature built into ZFS called the "hotspares" feature which allows a sysadmin to identify those drives available as spares which can be swapped out in the event of a drive failure in a storage pool. If an appropriate flag is set in the feature, the "hot spare" drive can even be swapped automatically to replace the failed drive. Or, alternatively, a spare drive can be swapped manually if the sysadmin detects a failing drive that is reported as irreparable. </p>
<p>Hot spares can be designated in the ZFS storage pool in two separate ways:</p>
<blockquote>When the pool is created using the <strong>zpool create</strong> command, and</blockquote>
<blockquote>After the pool is created using the <strong>zpool add </strong>command</blockquote>
<p>Before beginning to create the ZFS storage pool and identifying the spares, I want to list the available drives in my Debian 10 Linux VM in VirtualBox 6.0. I can accomplish this in the Terminal using the command:</p>
<figure class="m_original">
<img src="https://teletype.in/files/ed/1b/ed1bcab0-dc11-4c42-8920-a8f05e14ee7d.png" width="827" />
</figure>
<p>Here, we see that I have 7 total drives apart from the primary drive, /dev/sda, used by Debian 10 Linux. These drives are listed above and range from /dev/sdb ... to /dev/sdh.</p>
<p>Therefore, I plan to create a ZFS storage pool called <strong>trinity</strong> with a mirror of two drives: <strong>sdb</strong> and <strong>sdc</strong>; and hot spares which I will identify as <strong>sdd</strong> and <strong>sde</strong>. </p>
<p>The diagram below illustrates the new ZFS mirrored storage pool with the two spares identified. In addition, I have taken the liberty of recreating the trinity/data, .../apps, and .../datapioneer datasets:</p>
<figure class="m_original">
<img src="https://teletype.in/files/ab/b9/abb9c8d6-1227-4346-bb15-48a4c097651b.png" width="1044" />
</figure>
<p> As reported above using the <strong>zpool status</strong> or <strong>zpool status trinity</strong> command, the two mirror drives sdb & sdc are ONLINE and I have identified two AVAIL spares sdd & sde. If I want to remove one of the hotspares and replace it with another drive, I can do this using the <strong>zpool remove</strong> command:</p>
<figure class="m_original">
<img src="https://teletype.in/files/1a/92/1a924edb-dc78-4297-bd2f-6b048519f43d.png" width="699" />
</figure>
<p>Since the spare drive was not INUSE but was AVAIL instead, we were able to remove it. Had it been INUSE, this wouldn't be allowed. Instead, the drive would have to be manually taken OFFLINE first, then removed. <em>One important thing to keep in mind here is that drives identified as spares in the system must have a drive capacity >= largest drive in the storage pool. </em>Let's add another spare to the storage pool by replacing the drive sdd that we previously removed. This is accomplished by running the <strong>zpool add trinity spare</strong> command and inserting the <em>sdd</em> drive designated as the replacement:</p>
<figure class="m_original">
<img src="https://teletype.in/files/cf/95/cf95afb2-bf8e-4682-a1e9-a01cfa704051.png" width="718" />
</figure>
<p>As an aside, if at anytime you wish to determine the health of a ZFS storage pool, you can either run the zpool trinity status command or zpool trinity status -x command. The latter will actually list the health so you don't have to look at the STATE category of the status screen: </p>
<p>root@debian-10-desktop-vm:/trinity# <strong>zpool status trinity -x</strong><br /><em>pool 'trinity' is healthy</em></p>
<p>Now, if an ONLINE drive fails, the zpool will move into a DEGRADED state. The failed drive can then be replaced manually by the sysadmin or, if the sysadmin has set the <strong>autoreplace=on</strong> in the property of the zpool, the failed drive will be automatically replaced by ZFS. The command to set this property in this example is:</p>
<p>root@debian-10-desktop-vm:/trinity# <strong>zpool set autoreplace=on trinity</strong></p>
<p>We can simulate the failure of one of our drives by manually taking it OFFLINE. So, let's pretend that drive sdb fails by manually OFFLINE'ing it:</p>
<figure class="m_original">
<img src="https://teletype.in/files/03/80/0380deee-e63f-4983-ad57-a1b51c6b033a.png" width="845" />
</figure>
<p>Note here that sdb now shows DEGRADED in the status and the overall status of the Zpool Trinity is in a DEGRADED State as well. If this had been a real world scenario where a drive failure actually occurred instead of simulating the failure by OFFLINE'ing the drive, one of the AVAIL spares would have replaced the failed drive, sdb, with either sde or sdd, reducing the AVAIL spares by one and bringing the Zpool back to ONLINE status. I can either ONLINE the device, sdb, since I know it to be a good drive, or I can simulate replacing this drive with one of the spares but doing so through a manual rather than automatic process. I will demonstrate the latter process using the <strong>zpool replace</strong> command:</p>
<figure class="m_original">
<img src="https://teletype.in/files/83/80/83804da7-9537-43f6-8d49-92da34368329.png" width="944" />
</figure>
<p>Now, the current status after running the zpool replace trinity sdb sde command to replace the failed sdb drive with the hotspare sde indicates that spare sde is currently INUSE rather than AVAIL and sde is ONLINE. </p>
<p>And, finally, if we detach the failed (OFFLINE) drive using the <strong>zpool detach trinity sdb</strong> command, the status of the Zpool storage pool should be returned to ONLINE in healthy status:</p>
<figure class="m_original">
<img src="https://teletype.in/files/a4/cf/a4cf9966-ebba-4278-881b-312c03535cb8.png" width="946" />
</figure>
<p>Note, now the AVAIL spares has been reduced to sdd only and sde has replaced the failed sdb drive which was formerly OFFLINE. The drive was resilvered with 507K bytes with 0 errors returned and this process was accomplished on Sat Dec 28 2019 at 16:04:34 Local time. </p>
<p>The last thing that we can do here before wrapping up this article is to add another drive to the "hot spares" list. We know from the previous command that we ran that we have drive sdf available for this purpose. So, let's add this drive as a spare. Since the drive, sdf, is being added after the ZFS storage pool was created, we can use the <strong>zpool add trinity spare sdf</strong> command to add this drive as an AVAIL spare:</p>
<figure class="m_original">
<img src="https://teletype.in/files/ac/e6/ace6e5d0-fedb-405d-bb8f-77d82e53fde0.png" width="942" />
</figure>
<p>Now, we have two spares AVAIL instead of the single drive and our ZFS storage pool is healthy once again.</p>
<p><a href="https://inlinuxveritas.com/Sk68PBb1U?cda=" target="_blank">Return to beginning of the article - Part 1</a></p>
datapioneer:HkX83eHkLSetting Up Quotas & Reservations in OpenZFS in Linux2019-12-28T17:02:41.964Z2019-12-28T23:17:58.793Z<img src="https://teletype.in/files/60/6e/606ea314-7309-45d1-a54f-e671c882d28b.png">ZFS supports quotas and reservations at the filesystem level. Quotas in ZFS set limits on the amount of space that a ZFS filesystem can use. Reservations in ZFS are used to guarantee a certain amount of space is available to the filesystem for use for apps and other objects in ZFS. Both quotas and reservations apply to the dataset the limits are set on and any descendants of that dataset.
<p>ZFS supports quotas and reservations at the filesystem level. Quotas in ZFS set limits on the amount of space that a ZFS filesystem can use. Reservations in ZFS are used to guarantee a certain amount of space is available to the filesystem for use for apps and other objects in ZFS. Both quotas and reservations apply to the dataset the limits are set on and any descendants of that dataset.</p>
<p>Primarily, quotas are set to limit the amount of space that a particular user of the system can consume so that no one user maximizes the Zpool in the dataset. In this article, I have setup two users in the Linux system: <strong>datapioneer</strong> (myself as the primary user) and <strong>dante</strong> (a secondary user of the Linux system) upon which OpenZFS on Linux has been installed.</p>
<p>Let's look at the zpool that I currently have setup in Linux for this example. Running both <strong>zpool status</strong> and <strong>zfs list</strong> commands in Debian 10 Linux produce the following profile of the ZFS filesystem:</p>
<figure class="m_original">
<img src="https://teletype.in/files/60/6e/606ea314-7309-45d1-a54f-e671c882d28b.png" width="849" />
</figure>
<p> Here, I have a RAIDZ of 3 SCSI drives with a logs mirror and two-drive SSD Zil SLOG. All drives are ONLINE and the zfspool is also ONLINE and healthy. As shown, the zfspool/data dataset is using 12M of data out of a total space of 19.0G. This means that both datapioneer and dante have access to this total 19.0G of zfspool drive space across the drives. I would like to limit the user, dante, to 5G of drive space so he doesn't inadvertently fill up the Zpool with data. In addition, I would like to reserve 5G of the Zpool under the dataset zfspool/apps for applications. Here's how we can accomplish both.</p>
<p>To limit 5G of drive space as a quota for dante under zfspool/data, we can run the following command:</p>
<figure class="m_original">
<img src="https://teletype.in/files/d9/d6/d9d65fea-a87d-40fb-8a78-ef84e8c1d82c.png" width="900" />
</figure>
<p>Setting the quota for dante under the zfspool/data/dante dataset to 5G is performed using: <strong>zfs set quota=5G zfspool/data/dante</strong>. In the figure above, I ran a <strong>zfs list</strong> to look at the current listing of drive utilization and you can see that dante has the entire 19.0G available to him. After running the <em>zfs set quota</em> command, we can confirm that we have limited dante to 5G by running the <strong>zfs get quota zfspool/data/dante</strong> command showing a value of 5G for the property: <em>Quota</em>. Rerunning zfs list also shows that only 4.99G, ~ 5G, is currently available to dante. However, note that the user datapioneer still has full use of the zpool wherein the amount of 19.0G of space is showing as available. The user dante will only be able to fill up 5G of space under the zfspool/data/dante dataset and any attempts to add more data to the drives will be denied. Now, if we add a child dataset called <strong>ws</strong> under zfspool/data/dante, the quota for <strong>/zfspool/data/dante/ws</strong> inherits the 5G limitation:</p>
<figure class="m_original">
<img src="https://teletype.in/files/29/0b/290b5aa7-64c3-444e-913a-b2e50ad3a3bd.png" width="856" />
</figure>
<p>Now, let's set a 5G reservation for apps under /zfspool/apps. What this means is that the ZFS filesystem will set aside 5G of drive space for use by apps only and nothing else will be able to use it. This is different from quotas set up in the system since setting quotas allow drive space use up to a certain value whereas establishing a reservation tells the ZFS filesystem not to allow use of the reserved amount by any object in the system other than apps under <em>/zfspool/apps</em>. To set a reservation of 5G for apps dataset in ZFS, we can run the command:</p>
<figure class="m_original">
<img src="https://teletype.in/files/a0/56/a0566f3d-13d5-4264-bca3-24a65c82fcac.png" width="906" />
</figure>
<p>After setting the 5G reservation in ZFS for apps, we see that the amount USED for <em>zfspool</em> in the system has increased from 24.4M to 5.02G (which includes the 5G reservation for apps) with 14.0G AVAIL instead of 19.0G AVAIL prior to setting up the reservation. Also, note that the amount available for apps under <em>/zfspool/apps</em> remains at 19.0G which does not establish a quota of any kind since this amount AVAIL remains unchanged before and after the reservation was set. However, what should be realized here is that the amount of space AVAIL to dante remains at 5G (as established in the Quota we setup earlier) but, the total amount AVAIL for datapioneer has been reduced from 19.0G to 14.0G (the difference being 5G) which reflects the fact that 5G of space has been reserved for apps and is no longer available to any other object in the <em>zfspool</em> system. </p>
<p><a href="https://inlinuxveritas.com/HkV-xNH18?cda=" target="_blank">Designating "Hot Spares" in the Storage Pool in OpenZFS - Part 6</a></p>
datapioneer:rJOfqWQy8Logs Mirror, Cache, & Snapshots in OpenZFS Filesystem on Linux2019-12-27T23:15:25.279Z2019-12-31T03:01:15.235Z<img src="https://teletype.in/files/b8/eb/b8eb1f57-027e-4d8e-b1cb-1df617b4d363.png">Another big advantage to installing ZoL or OpenZFS filesystem on Linux is that as the sysadmin you can create a mirror of two SCSI drives in the filesystem containing your system logs and create a cache consisting of two SSD drives in the filesystem containing system cache information. The logs mirror helps to balance the load of the ZFS pool in the system and also helps to ensure that your log files are preserved in the event of RAIDZ failure. The cache is a part of the ARC (Adaptive Replacement Cache) system in OpenZFS and assists in rebuilding drives to restore your system if drives begin to fail. Read cache is referred to as L2ARC (Level 2 Adaptive Replacement Cache), synchronous write cache is ZIL (ZFS Intent Log), SLOG (Separate...
<p>Another big advantage to installing ZoL or OpenZFS filesystem on Linux is that as the sysadmin you can create a mirror of two SCSI drives in the filesystem containing your system logs and create a cache consisting of two SSD drives in the filesystem containing system cache information. The logs mirror helps to balance the load of the ZFS pool in the system and also helps to ensure that your log files are preserved in the event of RAIDZ failure. The cache is a part of the ARC (Adaptive Replacement Cache) system in OpenZFS and assists in rebuilding drives to restore your system if drives begin to fail. Read cache is referred to as L2ARC (Level 2 Adaptive Replacement Cache), synchronous write cache is ZIL (ZFS Intent Log), SLOG (Separate Log Device).</p>
<p>To prepare for the creation of the logs mirror, I added two additional 10G SCSI drives to the ZFS pool. These are designated as /dev/sde and /dev/sdf, respectively. Next, I ran the following command in the Terminal to add the logs mirror:</p>
<figure class="m_original">
<img src="https://teletype.in/files/b8/eb/b8eb1f57-027e-4d8e-b1cb-1df617b4d363.png" width="1035" />
</figure>
<p>Creating the logs mirror pretty much guarantees preservation of logs information in OpenZFS in the event of drive failure. Creating the logs mirror is just the first step in creating what is referred to as the ZIL, SLOG which is a fast persistent write cache for ZFS writes to disk. Note I said creating the logs mirror is the first step. Now that we have created the logs mirror in ZFS, the second step is to create what is known as the ZFS cache (a part of the L2ARC - Level 2 Adaptive Replacement Cache). To create the cache in the system, I added two high speed SSD drives /dev/sdg and /dev/sdh and then ran the following command:</p>
<p># <strong>zpool add zspool cache /dev/sdg /dev/sdh</strong></p>
<p>To check the status of the zpool at this point, we can rerun the <strong>zpool status</strong> command:</p>
<figure class="m_original">
<img src="https://teletype.in/files/ae/81/ae81ecd9-d52b-455a-ba79-77a92e27c1bb.png" width="559" />
</figure>
<p>The size of the disks that contain the logs and cache need to be determined based on the performance of your system. Monitoring this will give you a better indication of the apparent size these drives need to be. Using high speed SSDs or NVMe M.2 drives for the cache rather than traditional SCSI drives is a good idea especially if you don't want to start running into performance bottlenecks when cache information is being written to these cache drives during OpenZFS resilvering of replacement drives.</p>
<p>Combining the ZIL and Separate Log Device (SLOG) in the ZIL, SLOG configuration that you see above, you are essentially balancing the load on your system using OpenZFS.</p>
<p>ZFS Snapshots are essentially instances in time of the ZFS filesystem across the entire pool or datasets within the pool that you wish to capture. Snapshots in ZFS are read-only, immutable, great for backups since you can backup the snapshot rather than the files, and snapshots within the same host are not backups. Snapshots can be exported from one host to another host or within the same host. When creating snapshots, it is important to note that these snapshots in Linux will be represented in the system as hidden directories under the dataset wherein they are created with the snapshots being placed in those hidden directories. Files within the snapshots that are deleted may be restored using a rollback procedure which I will demonstrate below. Let's create a snapshot of the /zfspool/data/apps dataset:</p>
<figure class="m_original">
<img src="https://teletype.in/files/a7/97/a7975e3f-7ce0-4b9b-8f61-a4ad2f68c938.png" width="1021" />
</figure>
<p>The command: # <strong>zfs snapshot zfspool/apps@271220191653</strong> which creates a snapshot of the apps dataset represents an instance in time at 16:53 on 27 December 2019 of the zfspool/apps dataset. </p>
<p>After creating this snapshot, I used vim to create a file called <strong>file1</strong> which I placed in the <strong>/zfspool/apps</strong> directory where the snapshot was created:</p>
<p>root@debian-10-desktop-vm:/zfspool/apps# ls -lh<br />total 1.0K<br />-rw-r--r-- 1 root root 15 Dec 27 16:55 <strong>file1</strong></p>
<p>Now, if I rollback the snapshot using the ZFS rollback command, this file should no longer exist since it was created after the snapshot <strong>zfspool/apps@271220191653 </strong>was created:</p>
<p>root@debian-10-desktop-vm:/zfspool/apps# <strong>zfs rollback zfspool/apps@271220191653</strong><br />root@debian-10-desktop-vm:/zfspool/apps# ls -lh<br /><strong>total 0</strong><br />root@debian-10-desktop-vm:/zfspool/apps# <br /></p>
<p>and, indeed, the rollback has eliminated the file which was created earlier. Snapshots allow you to create a mark in time for your system. If you wish to list out the snapshots that have been created in the ZFS filesystem, you can accomplish this using the command:</p>
<figure class="m_original">
<img src="https://teletype.in/files/39/42/39428054-ea07-43bb-9ec0-1d6499d5dd2c.png" width="704" />
</figure>
<p>These snapshots do take up space on your system, but, as you can see here, this snapshot is only using 17.3K in the system.</p>
<p>And, finally, if you want to move a zpool to another machine, you need to get out of the zpool you wish to move, then use the following command:</p>
<p>root@debian-10-desktop-vm:/zfspool/apps# pwd<br />/zfspool/apps<br />root@debian-10-desktop-vm:/zfspool/apps# cd ../../<br />root@debian-10-desktop-vm:/# pwd<br />/<br />root@debian-10-desktop-vm:/# <strong>zpool export zfspool</strong><br />root@debian-10-desktop-vm:/# zpool status<br /><strong>no pools available</strong><br />root@debian-10-desktop-vm:/#</p>
<p>As you can see above, I moved up the directory tree two levels to the / directory, then ran the command which is highlighted. After running the <strong>zpool status</strong> command, you see that no pools are available. This is because the zpool has been exported. I did not use the switch on the export command to direct the export to a specific directory in Linux, but you probably should do this so you know where you put it. In this example, I let it place the exported zpool in the default location because I am going to turn right around and import the zpool back to its original location. I do this using the ZFS import command like this:</p>
<figure class="m_original">
<img src="https://teletype.in/files/df/1b/df1b62c4-ba19-4dfc-a874-84576c1114c7.png" width="564" />
</figure>
<p>The zpool zfspool has been returned and running a status on the pool returns good status as before. </p>
<p><a href="https://inlinuxveritas.com/HkX83eHkL?cda=" target="_blank">Setting up Quotas & Reservations in OpenZFS - Part 5</a></p>
<p><br /></p>
datapioneer:Sk88mPzyUInvestigating RAIDZ in Debian 10 "Buster" Linux2019-12-26T20:17:26.342Z2019-12-29T03:56:05.786Z<img src="https://teletype.in/files/43/ee/43eeebe9-6aca-4efd-8276-efcd4574e1a5.png">Now that we have looked at implementing OpenZFS on Linux in Debian 10 Linux and created zfs pool mirrors using OpenZFS as well as created and accessed ZFS datasets on the system, let's turn our attention to implementing RAID in OpenZFS. How does implementing RAID in OpenZFS compare to traditional RAID solutions? Is there a one-to-one correlation between RAIDZ and traditional RAID? Are there advantages to running RAIDZ rather than the traditional RAID solutions?
<p>Now that we have looked at <a href="https://inlinuxveritas.com/Sk68PBb1U?cda=" target="_blank">implementing OpenZFS on Linux in Debian 10 Linux</a> and created zfs pool mirrors using OpenZFS as well as created and accessed ZFS datasets on the system, let's turn our attention to implementing RAID in OpenZFS. How does implementing RAID in OpenZFS compare to traditional RAID solutions? Is there a one-to-one correlation between RAIDZ and traditional RAID? Are there advantages to running RAIDZ rather than the traditional RAID solutions? </p>
<p>RAIDZ in OpenZFS is roughly equivalent to RAID-5 in the traditional RAID world which requires at least 4 drives with one drive serving as a parity drive. RAIDZ (RAID-5) can survive the loss of one drive and still function in the RAID. RAIDZ2 is roughly equivalent to RAID-6 which requires a total of 6 drives with two drives serving as parity drives. RAIDZ2 (RAID-6) can survive the loss of two drives and still function within the RAID-array. RAIDZ3 has no equivalent in traditional RAID. </p>
<p>The first thing that we need to do is to destroy the current zpool arrangement that we established when I showed you how to implement OpenZFS on Debian 10 "Buster" Linux and set up ZFS pools or mirrors in the original article shown in paragraph 1 above. The process of destroying the ZFS pools is irreversible, so if there is any data that is important to you or you want to go back to this OpenZFS configuration, then you should perform a backup by taking a snapshot of the system. We are using VirtualBox 6.0 Manager to create this OpenZFS environment in Debian 10 Linux, so it is very simple to create a snapshot. If you don't know how to create a snapshot of your VM (similar to creating a Restore Point in Windows), then <a href="https://www.techrepublic.com/article/how-to-use-snapshots-in-virtualbox/" target="_blank">consult this resource</a> on how to accomplish this prior to moving forward.</p>
<p>Before destroying the current ZFS pool configuration, I suggest shutting down the VM and adding some additional SCSI drives. In the example below, I have added three more virtual SCSI drives to the VM. These are /dev/sdf, /dev/sdg, and /dev/sdh for a total of seven SCSI drives in the system. </p>
<figure class="m_original">
<img src="https://teletype.in/files/43/ee/43eeebe9-6aca-4efd-8276-efcd4574e1a5.png" width="719" />
</figure>
<p>Now, restart the VM and return to the Terminal in Debian 10. To destroy the current OpenZFS configuration we established in Linux, obtain elevated privileges in the system by becoming root, then run the following command as shown in the diagram below:</p>
<figure class="m_original">
<img src="https://teletype.in/files/ac/2a/ac2a1af9-4820-4a43-a518-332b207098ab.png" width="587" />
</figure>
<p>The <strong>zpool destroy zfspool </strong>command is akin to running <em>rm -Rf <directory name></em> in Linux as this command will recursively remove all ZFS pools and datasets that currently reside on the system. Rerunning the <strong>zpool list</strong> command afterward confirms that we no longer have any OpenZFS pools or datasets present in the Linux system. It is worth mentioning here that this command not only removes all OpenZFS pools and datasets, but it also removes any snapshots that may have been taken in ZFS as well as any exports you may have performed and the zfs mountpoints that were created. Therefore, backups are critical at this point as reversing this process is impossible.</p>
<p>So, what is RAIDZ? Essentially, RAIDZ is the implementation of RAID in OpenZFS which is roughly equivalent to RAID-5 using traditional software/hardware RAID in Linux. One primary distinction between RAIDZ and RAID-5 that I would like to mention at this point is that unlike traditional RAID-5, RAIDZ does not read block-by-block when it rebuilds a drive that may be lost in the RAID or that is manually replaced in the RAID prior to failure. Rather, RAIDZ only looks at the data that resides on the SCSI drive(s) that are rebuilt and writes back to the replaced drive(s) only the data not a block-by-block restoration of the drive(s) itself. When OpenZFS restores a RAID drive, it reports that the drive has been "resilvered". We'll see this when I show you how to implement RAIDZ going forward in this article.</p>
<p>To create a RAIDZ or RAIDZ1 in Linux, I need to run the following command as root, then I can check the status of the newly-created RAIDZ running the subsequent command we've run previously. This process will take a few seconds longer to run since the RAIDZ takes a little more work. Here we are creating an equivalent to a traditional RAID-5 using this one OpenZFS command and attaching three SCSI drives in the process. This RAIDZ will allow us to survive the loss of one drive without losing the RAID:</p>
<figure class="m_original">
<img src="https://teletype.in/files/7f/69/7f698274-518a-4ed3-a018-a2b91a5d45a9.png" width="938" />
</figure>
<p>Now, if I want to add an additional RAIDZ pool under zfspool I can rerun the previous command replacing the "create" option with "add" and changing /dev/sdb, /dev/sdc, and /dev/sdd with /dev/sde, /dev/sdf, and /dev/sdg, respectively. See the diagram below:</p>
<figure class="m_original">
<img src="https://teletype.in/files/d5/95/d59550f9-542a-43de-89f3-c62b6b01ed65.png" width="890" />
</figure>
<p>Running a status of the zpool configuration now shows that we have created two RAIDZ mirrors: raidz1-0 and raidz1-1 in the system. The first RAIDZ comprises drives sdb through sdd and the second RAIDZ comprises drives sde through sdg. All of these RAID mirrors are ONLINE and no known errors have been detected in the drives. </p>
<p>If you recall from the <a href="https://inlinuxveritas.com/Sk68PBb1U?cda=" target="_blank">previous article on implementing OpenZFS in Linux</a>, we set all the SCSI drives to a 10G capacity. Let's investigate how much drive capacity we have usable among the six 10G-capacity SCSI drives that we have implemented in this scheme:</p>
<figure class="m_original">
<img src="https://teletype.in/files/12/21/12215764-ca15-4afa-bffd-e68e891eb638.png" width="554" />
</figure>
<p>From the diagram above, we see that OpenZFS is reporting a total of 38.1G of usable drive space out of a total of 60G. This is roughly 2/3rds of the available drive capacity among the six SCSI drives. We are losing 1/3rd of the capacity or roughly two SCSI drives to parity. In this RAIDZ configuration, we can survive the loss of a single drive and survive a system-wide data loss. We need to recreate the datasets /zfspool/apps, /zfspool/data, and /zfspool/data/datapioneer that we had previously. This is accomplished in the Terminal like so:</p>
<figure class="m_original">
<img src="https://teletype.in/files/52/ac/52acae62-9e17-4460-96a6-ec485ff58087.png" width="827" />
</figure>
<p>We also need to reacquire the datapioneer dataset we just created which will require us to change ownership of /zfspool/data/datapioneer to datapioneer and group ownership to datapioneer as well as changing the permissions to 755 on this dataset (mounted directory for the zfspool in Linux). See the diagram below:</p>
<figure class="m_original">
<img src="https://teletype.in/files/65/76/65767cb1-b2db-454d-9b3d-a4b64437c126.png" width="887" />
</figure>
<p>Running ls -ld against the dataset confirms this has been accomplished. I have written a Bash script which writes data to the zpool we created and made accessible, namely, /zfspool/data/datapioneer. The bash script looks like the following:</p>
<figure class="m_original">
<img src="https://teletype.in/files/2d/c9/2dc99a0a-cafa-4e40-8660-3f1d38f5a3cf.png" width="737" />
</figure>
<p>I called this bash script, data.sh., added the executable bit to the file, then ran the script which added 3M X 4 (over 4 tests) of data to the zfspool dataset. To confirm this, we can run the df -kh command in the Terminal. This reveals:</p>
<figure class="m_original">
<img src="https://teletype.in/files/0b/8a/0b8aa8f6-9c22-4ccd-a272-80b876f2e43c.png" width="845" />
</figure>
<p>which demonstrates successful writing of ~12M (13M shown) of data to the dataset. Similarly, we can look at this in the GUI by bringing up the File Manager and taking a Properties sheet on the combined files test through test4 that were created in running the bash script:</p>
<figure class="m_original">
<img src="https://teletype.in/files/93/a2/93a2a3fd-b1cf-489a-9360-3eb72356ac0c.png" width="772" />
</figure>
<p>So, now, let's look at some real world examples of how RAIDZ works. If we return to take a look at the status of the current RAIDZ, we see the following output:</p>
<figure class="m_original">
<img src="https://teletype.in/files/8a/30/8a303663-6be8-4d22-9490-98a29eb92648.png" width="745" />
</figure>
<p>Let's assume that we, as sysadmin, see a bunch of CKSUMs starting to show up on drive <strong>/dev/sdc</strong> indicating a potential failure in that drive. We can take one of our spare drives, <strong>/dev/sdh,</strong> and replace /dev/sdc with this spare. To accomplish this on-the-fly using OpenZFS, we run the following command in the Terminal:</p>
<figure class="m_original">
<img src="https://teletype.in/files/52/20/5220c69a-cbc2-4bea-b038-30b9ee833eec.png" width="1050" />
</figure>
<p>In the command that we ran in the Terminal, we referenced <strong>/dev/sdc</strong> as the drive to be replaced and <strong>/dev/sdh</strong> as the drive replacing it. Note, under zfspool / raidz1-0 drive <strong>sdc</strong> has been replaced by the spare drive, <strong>sdh</strong>, which is showing up as ONLINE, and the comment following "scan:" indicated the zfspool has been "resilvered" which as I mentioned earlier is the equivalent to a RAID rebuilding of the replaced drive. Note, also, that this process took only 1 second to rebuild since unlike traditional RAID-5, RAIDZ looks only at the 12M of data that it needs to rewrite to the spare drive rather than rebuilding the entire 10G drive block-by-block.</p>
<p>So, now let's look at another real world scenario in which a SCSI drive in the RAIDZ zfspool actually fails. I have replaced /dev/sdh with /dev/sdc once again and then simulated a drive failure in drive /dev/sdc using the following command in the Terminal and outputting the status following the running of the command:</p>
<figure class="m_original">
<img src="https://teletype.in/files/6c/e0/6ce0d61b-404e-498e-a9d9-8683a3a67494.png" width="976" />
</figure>
<p>Taking /dev/sdc OFFLINE results in a degraded zfspool which is shown in the "state" of the pool as well as an OFFLINE status for sdc under raidz1-0 with a STATE of "degraded". The "status" reported shows one or more devices has been taken offline by the administrator (simulating the failure) and that "sufficient replicas exist for the pool to continue functioning in a degraded state." The recommended "action" is to "Online the device using 'zpool online' or replace the device with 'zpool replace'."</p>
<p>In this example, we're going to ONLINE the drive /dev/sdc, by running the appropriate command shown below, then rerunning a status to show the current state of the zpool in a non-DEGRADED state:</p>
<figure class="m_original">
<img src="https://teletype.in/files/62/50/6250c593-cf0d-4b5a-8ee6-ae7f943776f6.png" width="974" />
</figure>
<p>The drive, /dev/sdc, has been resilvered and the entire zfspool has been returned to an ONLINE status from DEGRADED status reported earlier.</p>
<p>Now, notice that when we run <strong>zfs list</strong> again in the Terminal, I, as datapioneer, have access to 38.1G of usable drive space across the six SCSI drives and, could theoretically fill that entire space on my own. </p>
<figure class="m_original">
<img src="https://teletype.in/files/f3/e3/f3e36245-969a-4b5f-8d2c-a877badbbbc9.png" width="845" />
</figure>
<p>But, in OpenZFS, there is a way to limit the amount of this usable space that a single user can consume. This is referred to as establishing a QUOTA on the pool. If I want to limit my quota of usable drive space to 100M rather than the entire 38.1G, I can run the following command in the Terminal:</p>
<p># <strong>zfs set quota=100m zfspool/data/datapioneer</strong></p>
<p>Then, we can confirm this by running the following command and looking at the output in the Terminal:</p>
<figure class="m_original">
<img src="https://teletype.in/files/75/cf/75cfeafd-9405-4c0e-8460-172c4dfec688.png" width="1027" />
</figure>
<p>The quota assigned to datapioneer is listed as 100M, not the full pool of 38.1G. So, if I start creating files that begin to fill up the pool beyond 100M, future actions to create files will be disallowed until action is taken to reduce the amount of used space across the six drives below the quota assigned to me. Granted, 100M of allowed space in today's world seems totally preposterous, but you get the point. </p>
<p>Another good example here would be to run a zfs command that would allow the administrator to reserve 100M of usable space for apps on the pool. Unlike quotas, this is considered a <strong>reservation</strong> of space that is to be used by apps only so that no one else can use it. To perform this action in OpenZFS against the pool, we can run the following command in the Terminal:</p>
<figure class="m_original">
<img src="https://teletype.in/files/8d/a1/8da1930d-cb27-446d-b425-f7122e57345c.png" width="1036" />
</figure>
<p>So, by looking at the before-and-after listing of zfs we see that initially there was 38.1G available space in zfspool. After running the reservation using the command shown to reserve 100M of space for apps, the amount of available space in zfspool has been effectively reduced to 38.0G so that others cannot use the difference of 0.1G or 100M. </p>
<p>And, finally, the last thing that I would like to show as an example of a real world scenario using OpenZFS is to simulate a process in OpenZFS that is similar to the process that can be performed in Linux but which is much more difficult to perform in the latter case. What I'm referring to here is this. If you start noticing a drive might be failing or data might begin becoming corrupt on the drive, in Linux you would have to take the drive OFFLINE by <strong>umounting</strong> the drive (and this could be in a Linux RAID scenario), then run <strong>fsck</strong> against the drive, getting a status, then returning the drive ONLINE by <strong>mounting</strong> it if you find there are no errors or if errors have been corrected. One of the beauties of OpenZFS is that a sysadmin can check a drive or the pool of drives in RAIDZ, for example, without having to OFFLINE a particular drive. This process is referred to as a <strong>SCRUB</strong> in OpenZFS and it can be performed at anytime while the ZFS pool is functioning or on a schedule using Cron. To perform this action manually, you can run the following command in the Terminal:</p>
<figure class="m_original">
<img src="https://teletype.in/files/f2/30/f230e592-67d8-46e2-bc0f-0ca26054dd16.png" width="996" />
</figure>
<p>Running the command as shown above performed the scrub scan against the zfspool and it reports that 0B of data was repaired in 1 second. If lots of data were repaired instead, this would be an indication of corruption occurring in the pool and the sysadmin would be alerted to a potentially failing drive that needed action taken to replace.</p>
<p>In a future article, I will look at the implementation of RAIDZ2 and RAIDZ3 and its implications.</p>
<p><a href="https://inlinuxveritas.com/rJOfqWQy8?cda=" target="_blank">Logs, Mirrors, Cache, & Snapshots in OpenZFS in Linux - Part 4</a></p>
datapioneer:Sk68PBb1UInstalling and Using OpenZFS on Debian 10 "Buster" Linux2019-12-26T05:05:36.047Z2019-12-29T03:26:03.912Z<img src="https://teletype.in/files/a6/76/a6762812-6f3c-4bb5-aaac-4eb690c161a0.png">I am running Debian 10 "Buster" Linux in Virtual Box 6.0 Manager on my Win10 Pro Main PC using the debian10-1.0-amd64-netinst.iso file which I downloaded from the Debian Linux download page. This distro was originally installed as a VM using the ext4 filesystem for the primary partition represented as /dev/sda1 in the system. I wanted to experiment with using ZFS (ZetaByte File System) which was originally developed by Sun Microsystems and published under the CDDL license in 2005 as part of the OpenSolaris operating system. I further wanted to investigate this filesystem over others that are traditionally used in Linux, such as ext3/ext4/btrfs because ZFS is known for two specific reasons: (1) It stores large files in compressed format...
<p>I am running Debian 10 "Buster" Linux in Virtual Box 6.0 Manager on my Win10 Pro Main PC using the debian10-1.0-amd64-netinst.iso file which I downloaded from the <a href="https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/" target="_blank">Debian Linux download page</a>. This distro was originally installed as a VM using the ext4 filesystem for the primary partition represented as /dev/sda1 in the system. I wanted to experiment with using ZFS (ZetaByte File System) which was originally developed by Sun Microsystems and published under the CDDL license in 2005 as part of the OpenSolaris operating system. I further wanted to investigate this filesystem over others that are traditionally used in Linux, such as ext3/ext4/btrfs because ZFS is known for two specific reasons: (1) It stores large files in compressed format, and, (2) it decouples the filesystem from the hardware or the platform on which it is running. In my specific case, I'm running Debian 10 in a Virtual Machine rather than on bare metal and I'm running Linux, not Windows, MacOS, or BSD. </p>
<p>The implementation of ZFS that I am undertaking, however, isn't ZFS but, instead, is OpenZFS. I chose OpenZFS because unlike ZFS which is a proprietary filesystem developed under Oracle, the latter is opensource and community-supported and licensed under its own opensource licensing. OpenZFS has been ported from FreeBSD to Linux in support of the ZoL (ZFS on Linux) installation on Linux distros like Debian, Arch, Fedora, Gentoo, OpenSuse, RHEL and CentOS, and Ubuntu.</p>
<p>My OpenZFS on Debian 10 "Buster" Linux project was performed by following the steps below:</p>
<p>Step 1: Configure Debian 10 "Buster" Linux by setting it up in Oracle Virtual Box 6.0 Manager running in Windows 10 Pro and installing the operating system using the default ext4 filesystem. Apply all system updates prior to moving to Step 2, then shutdown the system.</p>
<p>Step 2: Add two VHD SCSI disks in the system designated as /dev/sdb and /dev/sdc, then restart the VM. See the diagram below which shows how I added the SCSI Controller in the Storage module, then added the two VHD SCSI Virtual Hard Drives on the SCSI Controller in the system:</p>
<figure class="m_column">
<img src="https://teletype.in/files/a6/76/a6762812-6f3c-4bb5-aaac-4eb690c161a0.png" width="718" />
</figure>
<p>Step 3: For Debian Buster Linux, the ZFS packages are included in the contrib repository. In my case, I used the backports repository which typically contain updated releases of these ZFS packages. I added the backports repository in Debian 10 Buster using the following commands in the Linux Terminal:</p>
<p># <strong>vi /etc/apt/sources.list.d/buster-backports.list</strong><br />deb http://deb.debian.org/debian buster-backports main contrib<br />deb-src http://deb.debian.org/debian buster-backports main contrib<br /><br /># <strong>vi /etc/apt/preferences.d/90_zfs</strong><br />Package: libnvpair1linux libuutil1linux libzfs2linux libzpool2linux spl-dkms zfs-dkms zfs-test zfsutils-linux zfsutils-linux-dev zfs-zed<br />Pin: release n=buster-backports<br />Pin-Priority: 990</p>
<p></p>
<p>Step 4: Run a system update to refresh the repositories using</p>
<p># apt update</p>
<p>and then install the kernel headers and associated dependencies using</p>
<p># apt install --yes dpkg-dev linux-headers-$(uname -r) linux-image-amd64<br /></p>
<p>And, finally, install the ZFS packages by running</p>
<p># apt-get install zfs-dkms zfsutils-linux</p>
<p>Step 5: My Debian 10 Buster Linux system was unable to find the <strong>zpool</strong> command when I started to execute the commands in the Linux Terminal to setup the two SCSI drives as a ZFS mirror pool. So, I had to add the path to the zfs commands so BASH would recognize them. I ran the command:</p>
<p># whereis zpool</p>
<p>which returned the location to the command as</p>
<p>/usr/sbin/zpool</p>
<p>Therefore, to allow my Linux system to find this command as well as other zfs commands by default, I modified the ~/.bashrc file and added the following line at the bottom of that file before saving it, then restarting the Terminal:</p>
<p><strong># adding the path to the zpool command in Linux<br />export PATH=$PATH:/usr/sbin/</strong></p>
<p>This makes the $PATH change persistent in the system. To prove this, I ran the command:</p>
<p>echo $PATH </p>
<p>in the Terminal and the following was returned:</p>
<p>/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:<strong>/usr/sbin/</strong></p>
<p>indicating that the $PATH has been extended to include the location for the zfs commands in the system making these commands recognizable by BASH by default.</p>
<p>Next, I created a ZFS Pool using the two SCSI drives /dev/sdb and /dev/sdc I created earlier into a ZFS mirror-0, using the command:</p>
<p># zpool create zfspool mirror /dev/sdb /dev/sdc</p>
<p>and checked the status of the ZFS pool I just created using the following <strong>zpool status</strong> command with <strong>-v</strong> switch, and then running the zfs list command to indicate the amount of pool space currently available and where it is mounted. I followed this up by running the <strong>df -kh</strong> command to have Linux show me the filesystem breakdown in the system by human-readable blocksize. See these commands listed out below:</p>
<figure class="m_original">
<img src="https://teletype.in/files/e4/89/e489c8b6-fef9-4344-a998-46e4e9b334f2.png" width="742" />
</figure>
<p>Note above that the zfspool is showing up as ONLINE state and both sdb and sdc along with the associated mirror-0 are ONLINE as well. No errors were detected in these drives. The <strong>zfs list </strong>command shows that 88.5k of data was written to the mirror for administrative tracking purposes leaving 9.3G out of 20G of total drive space (10GB X 2) as available for the zfspool mirrored drive. <em>Also, note that the mount of the zfspool at /zfspool as shown in the diagram above is persistent. It means that by implementing the OpenZFS filesystem using ZoL I am no longer required to update the /etc/fstab file in Linux each time I create a ZFS pool. Thus, the zfspool which was created is not managed by Linux at all. Instead, ZFS manages the pool of two drives, in this example, and automatically mounts the pool as mirror-0 (listing out the drives separately). </em></p>
<p>Step 6: Now, I want to expand my current zfspool of drives from its current 10G size to 20G by adding a new mirror of two drives designated as /dev/sdd and /dev/sde. To accomplish this, I need to stop the VirtualBox 6.0 Manager VM of Debian 10 Buster, then add two SCSI drives as VHD virtual drives which show up as /dev/sdd and /dev/sde in the Linux system. The process to add these two additional drives in VirtualBox 6.0 Manager is shown below:</p>
<figure class="m_original">
<img src="https://teletype.in/files/0f/5d/0f5dae0b-25f7-44c7-a974-db0f0cc9d1e7.png" width="713" />
</figure>
<p>Note, we now have two additional drives under the SCSI Controller: /dev/sdd and /dev/sde. Each of these drives were set to a size of 10G.</p>
<p>Next, I use the command:</p>
<p># zpool add zfspool mirror /dev/sdd /dev/sde</p>
<p>which adds the two SCSI drives to the existing zfspool as mirrored drives. If I then list out the zpool, run the Linux command that displays the existing mount of zfspool and its size, and obtain a current status of the zpool, you can see that I have effectively expanded the existing zfspool I originally created to a size of 20G (doubling the drive space). </p>
<figure class="m_original">
<img src="https://teletype.in/files/04/74/0474a8c4-9358-49c3-9f4b-4c3cd42d96a8.png" width="671" />
</figure>
<p>Now that I've created mirror-0 by combining /dev/sdb and /dev/sdc as well as mirror-1 by combining /dev/sdd and /dev/sde, I can no longer add more drives to either mirror. However, I can add additional drives by creating additional mirrors.</p>
<p>In ZFS, Pools are the equivalent of Disk Volumes in RAID and other disk-combining systems. Moreover, in ZFS, Datasets are the equivalent of data shares in those other systems. Thus, in the next step, I create my first dataset in ZFS underneath the zfspool mounted at /zfspool. </p>
<p>Step 7: To create a ZFS dataset called data under zfspool, I run the following command, then list out the status of the pool:</p>
<figure class="m_original">
<img src="https://teletype.in/files/a7/98/a7984236-40f4-4622-a72a-80d8faac66ee.png" width="771" />
</figure>
<p>This creates zfspool/data mounted at /zfspool/data automatically and 24k of administrative data was written to the pool to keep track of it. Now, if I wanted to add another ZFS Dataset called apps and add another ZFS Dataset called datapioneer underneath the data ZFS Dataset, I can do this in the Terminal as follows:</p>
<figure class="m_original">
<img src="https://teletype.in/files/fa/7f/fa7fbe99-4682-4737-9f9d-235b03b21cfa.png" width="901" />
</figure>
<p>Step 8: To gain access to the /zfspool/data/datapioneer dataset and permissions to write to it, I can perform the following commands in the Linux Terminal:</p>
<figure class="m_original">
<img src="https://teletype.in/files/b5/4e/b54e158b-58e4-4f86-97a4-ad6d11c68141.png" width="1028" />
</figure>
<p>Listing the storage looking at directories only, you can see that I now own the /zfspool/data/datapioneer directory and the group owner of this directory is also datapioneer with rwx, r-x, and r-x permissions. This is possible in the Linux Terminal since ZFS has a Posix layer in its ZoL implementation in Linux. </p>
<p><a href="https://inlinuxveritas.com/SyLKc3ZkL?cda=" target="_blank">Accessing The Mirror - Part 2</a></p>
datapioneer:SyLKc3ZkLAccessing an OpenZFS Mirror Created in Debian 10 Linux2019-12-26T05:03:29.944Z2019-12-28T22:54:38.269Z<img src="https://teletype.in/files/36/72/3672ba4d-2fb3-4cb5-a9f4-36b9778271d6.png">Now that we have created the ZFS Mirrors in the Linux system which point to four other SCSI drives of 10G capacity each, our two existing mirrors of two drives each are theoretically capable of accessing data of at least 20G in size. However, due to the overhead for keeping track of this data in the filesystem, our Linux system show a total data access space of around 19G.
<p>Now that we have created the ZFS Mirrors in the Linux system which point to four other SCSI drives of 10G capacity each, our two existing mirrors of two drives each are theoretically capable of accessing data of at least 20G in size. However, due to the overhead for keeping track of this data in the filesystem, our Linux system show a total data access space of around 19G. </p>
<p>If I open the Linux Terminal and run a listing of the storage using the ls command, then change directory to the zfspool to the subdirectory called datapioneer under the data subdirectory of zfspool, and then run the df -kh command, the output indicates the 19G blocksize that I mentioned above. </p>
<figure class="m_original">
<img src="https://teletype.in/files/36/72/3672ba4d-2fb3-4cb5-a9f4-36b9778271d6.png" width="1278" />
</figure>
<figure class="m_original">
<img src="https://teletype.in/files/38/0b/380b09e9-e579-4307-93f6-1f9759f108af.png" width="855" />
</figure>
<p>The total available space in the /zfspool/data/datapioneer dataset directory is 19G. This can be seen both in the Terminal and in File Manager as shown above. Now, since I changed ownership of this directory to datapioneer and set the permissions on the directory to 755, as datapioneer in the Linux system I should be able to touch this directory and create a file, name the file, add some content to it, and save the file under /zfspool/data/datapioneer with a total of 19G of available space in which to do so. This is demonstrated below in the Terminal:</p>
<figure class="m_original">
<img src="https://teletype.in/files/bb/fb/bbfb78b9-423b-40b0-a3b3-daaf4fab8d2a.png" width="1321" />
</figure>
<p>Here, a new file was created called newFile.txt containing the text shown in double quotes following the echo command which is then redirected as output to the file itself. Opening this file in File manager shows a clearer picture in the GUI:</p>
<figure class="m_original">
<img src="https://teletype.in/files/77/6b/776b4201-6628-4cf7-ae5a-adc02dd71bf2.png" width="889" />
</figure>
<p>Thus, we have created an additional 20G available ZFS pool of 4 SCSI drives of 10G capacity each that allow us to expand our available space across all four drives in the Linux system for storing our files which can be also be shared on the network.</p>
<p>If, in the future, additional storage space is needed, then all that is required is to install the additional SCSI drives, create a new mirror, and add them to the ZFS Pool expanding the total space across all drives in the Linux system. This is easily performed under OpenZFS. In this example I used small capacity SCSI drives, but I could just as easily add 1, 2, 3, or 4TB SCSI drives in a mirrored fashion to greatly enhance available storage and doing so is accomplished quite easily, as you can see, with the implementation of ZoL in Linux.</p>
<p><a href="https://inlinuxveritas.com/Sk88mPzyU?cda=" target="_blank">Investigating RAIDZ in Debian 10 "Buster" Linux - Part 3</a></p>
datapioneer:HkgGuWDsrPersonal Writing Blog2019-11-11T16:10:47.796Z2019-11-11T16:12:16.807Z@ Scripto (Veritas)
<p><a href="https://teletype.in/@scripto" target="_blank">@ Scripto (Veritas)</a></p>
datapioneer:Hk4eAxCKrLinux Unix Tech Channel2019-10-23T16:21:00.245Z2019-10-23T16:25:31.095Z<img src="https://teletype.in/files/4e/4e3d8f66-8d5f-4063-8822-45e2b0f03c97.png">The link to my YouTube Channel is: https://www.youtube.com/user/dlcalloway/
<p>The link to my YouTube Channel is: <a href="https://www.youtube.com/user/dlcalloway/videos?view_as=subscriber" target="_blank">https://www.youtube.com/user/dlcalloway/</a></p>
<figure class="m_retina">
<img src="https://teletype.in/files/4e/4e3d8f66-8d5f-4063-8822-45e2b0f03c97.png" width="960" />
</figure>
datapioneer:BJoISLXFSSetting Up a NAS Solution Running in Debian Linux2019-10-15T14:26:04.834Z2019-10-15T14:26:04.834ZRecently, I setup a secondary network-attached storage (NAS) solution at home using a virtual machine rather than a bare metal PC/server as a test platform. The process for setting this up is rather easy and anyone can do it. As I stated here, this is a secondary NAS solution since I already have a 5TB WDMyCloud Personal Cloud which I have had in place in my home running on my LAN now for several years.
<p>Recently, I setup a secondary network-attached storage (NAS) solution at home using a virtual machine rather than a bare metal PC/server as a test platform. The process for setting this up is rather easy and anyone can do it. As I stated here, this is a secondary NAS solution since I already have a 5TB WDMyCloud Personal Cloud which I have had in place in my home running on my LAN now for several years.</p>
<p>I have Virtualbox 6.0 Manager running on my Windows 10 Pro main PC from which I created my virtual environment of an application called OpenMediaVault. This application runs on Debian Linux and can be downloaded from <a href="https://www.openmediavault.org/?page_id=77" target="_blank">OpenMediaVault's download page</a>. There is a link on this page to the ISO images that you can select from to get started. In my particular test case in Virtualbox 6.0 Manager, I selected the 5.0.5 image. If, however, you are electing to install OpenMediaVault on a Raspberry Pi 3 or 4 device, then you will want to elect to download and install the Raspberry Pi Images onto an SD Card and insert it in the Pi to install and utilize your Raspberry Pi device for this purpose. Since this is a test project for me so I can see how OpenMediaVault works and decide whether I want to use it as a secondary NAS solution, I chose the former rather than the latter ISO image.</p>
<p>Rather than describe all the necessary steps that I used to download, install, and configure OpenMediaVault as a VM in Virtualbox 6.0 Manager on my Windows 10 Pro platform, I will, instead, point you to the video that walks you through the entire process. If you have questions about this process after watching the video, then please leave a comment below this article and I will attempt to answer any questions as I get to them.</p>
<figure class="m_column">
<iframe src="https://www.youtube.com/embed/VmMyZj_wpeM?autoplay=0&loop=0&mute=0"></iframe>
</figure>