On Thu, Jun 02, 2016 at 05:38:59AM -0400, Greg Freemyer wrote:

Hoping to get access to a laptop with a NVMe / M.2 storage subsystem later today. (a Alienware laptop purchased in March or April).

Note NVMe is not sata/scsi based. It's a different data bus.

Does anyone know what device it should create? /dev/sda is for devices under the overarching SCSI driver including those handled by libata. I don't know if NVMe is layered under the SCSI driver or not?

Nope, NVMe is a own technology, although it has a translation layer for _some_ SCSI ioctl()s. It doesn't create /dev/sd* device nodes as well but for example a /dev/nvme0n1p1. /dev/nvme0 is the 1st NVMe drive in your system, /dev/nvme0n1 is the 1st namespace (similar to a SCSI LUN) on that drive /dev/nvme0n1p1 is the 1st partition in that namespace.

I'll try the latest Tumbleweed boot media with it via attaching an external DVD.

Is this tech known to work with Tumbleweed?

Haven't tried with Tumbleweed but Leap 42.1 and SLES 12 SP2 Beta 2 work realiably, so Tumbleweed should as well, after all it has a fairly new kernel. For userspace support, there is nvme-cli if you want to play around with it, but usually that's not needed. If there are any problems please report them in Bugzilla and either assign them directly to me or at least put me in Cc so I'm aware of it. Byte, Johannes -- Johannes Thumshirn Storage jthumshirn@suse.de +49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Jun 2, 2016 at 8:42 AM, Andrei Borzenkov wrote:

On Thu, Jun 2, 2016 at 12:38 PM, Greg Freemyer wrote:

...

Hoping to get access to a laptop with a NVMe / M.2 storage subsystem later today. (a Alienware laptop purchased in March or April).

Is it NVMe pure, without any other mass media? This should be challenging for bootloader configuration. Looking forward for your bug reports :)

At least for now, I will have read only access to the laptop. It's a client's, but in my control for at least a little bit today. I saw it yesterday for the first time. It has a SSD and a rotating disk. Windows is installed on the SSD and that is what it boots from. I had a openSUSE 13.1 Live DVD with me, so I booted that up to verify I could see the SSD. Negative. But it boots Windows fine. I only looked with /dev/sd* type searches. Didn't know about /dev/nvme* (and I assume 13.1 has too old of a kernel for nvme anyways.)

...

Note NVMe is not sata/scsi based. It's a different data bus.

M.2 is form-factor; it can support PCIe or SATA (or even USB) so you need to check precisely what storage technology is used in your case.

How? What Linux commands? I'll burn the latest Tumbleweed ISO to test with. I did a physical inspection of the SSD (just a few screws to get the back off). It is definitely a M.2, but with a M Key notch. First time I've ever seen a M.2 with a M.Key notch. https://en.wikipedia.org/wiki/M.2#Form_factors_and_keying fyi: Accessing hard drives (including SSDs) on client on computers is a big part of my job, so I'm willing to use SuSEstudio to burn a custom DVD if that is needed for proper tool inclusion. Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Jun 2, 2016 at 3:56 PM, Andrei Borzenkov wrote:

02.06.2016 16:27, Greg Freemyer пишет:

...

Didn't know about /dev/nvme* (and I assume 13.1 has too old of a kernel for nvme anyways.)

NVMe should be supported starting with 3.3. Of course there could be specific hardware dependency.

...

...

...

Note NVMe is not sata/scsi based. It's a different data bus.

M.2 is form-factor; it can support PCIe or SATA (or even USB) so you need to check precisely what storage technology is used in your case.

How? What Linux commands? I'll burn the latest Tumbleweed ISO to test with.

Well, either you will see traditional SCSI devices or PCI devices. But looking at various reports, it may be necessary to fiddle with BIOS settings to expose NVMe devices in Linux.

Don't tell Richard, but I think I'm going to go crazy and buy a couple of bleeding edge devices to test with: === M.2 M Key device using AHCI protocol === Samsung XP941 SSD PCIe M Key 2280 MZHPU128HCGM-00000 $120 from Amazon === M.2 M Key device using NVMe protocol === Samsung PM951 128GB M.2 NGFF PCIe Gen3 x4, NVME Solid state drive SSD, OEM (2280) ( MZVLV128HCGR-00000 $74 from Amazon ==== Both of those should do well over 500 MB/sec in sequential read access situations. Do you know if a PCIe adapter card has to support AHCI vs NVMe, or does the PCIe card just connect the PCIe bus pins to the M.2 connector pins and could careless what protocol is in use? I've got a couple workstations with different X99 motherboards. Hopefully one or both supports booting from both of those SSDs. Regardless it will be an interesting learning experience. I simply don't know much about this class of SSD yet, and I really should now that prices are getting down to affordable. It looks to me that these devices will soon eliminate the 2.5" form factor SSDs with a SATA interface that I'm most used to seeing in laptops. (I have seen a number of mSATA and M.2 B-Keyed SSDs already, but the 2.5" form factor still dominates what I see in my work. The NVMe SSD in the Alienware is the first NVMe SSD I've come across.) Thanks Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Jun 02, 2016 at 04:21:22PM -0400, Greg Freemyer wrote:

On Thu, Jun 2, 2016 at 3:56 PM, Andrei Borzenkov wrote:

...

02.06.2016 16:27, Greg Freemyer пишет: === M.2 M Key device using AHCI protocol ===

Samsung XP941 SSD PCIe M Key 2280 MZHPU128HCGM-00000

AHCI is _not_ NVMe!!! AHCI is SATA. That's the trickery with the M.2 Slot. It can be SATA, USB or PCIe.

$120 from Amazon

JFTR I've made good experiences with the following Samsung M.2 NVMe: # nvme list Node Model Version Namepace Usage Format FW Rev ---------------- -------------------- -------- -------- -------------------------- ---------------- -------- /dev/nvme0n1 SAMSUNG MZVPV256HDGL 1.1 1 26,26 GB / 256,06 GB 512 B + 0 B BXW72H0Q A quick search show it @ ~125€.

=== M.2 M Key device using NVMe protocol ===

Samsung PM951 128GB M.2 NGFF PCIe Gen3 x4, NVME Solid state drive SSD, OEM (2280) ( MZVLV128HCGR-00000

$74 from Amazon ====

Both of those should do well over 500 MB/sec in sequential read access situations.

Do you know if a PCIe adapter card has to support AHCI vs NVMe, or does the PCIe card just connect the PCIe bus pins to the M.2 connector pins and could careless what protocol is in use?

I've got a couple workstations with different X99 motherboards. Hopefully one or both supports booting from both of those SSDs. Regardless it will be an interesting learning experience. I simply don't know much about this class of SSD yet, and I really should now that prices are getting down to affordable.

It looks to me that these devices will soon eliminate the 2.5" form factor SSDs with a SATA interface that I'm most used to seeing in laptops. (I have seen a number of mSATA and M.2 B-Keyed SSDs already, but the 2.5" form factor still dominates what I see in my work. The NVMe SSD in the Alienware is the first NVMe SSD I've come across.)

Thanks Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

-- Johannes Thumshirn Storage jthumshirn@suse.de +49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, Jun 03, 2016 at 10:48:05AM +0300, Andrei Borzenkov wrote:

On Fri, Jun 3, 2016 at 10:31 AM, Johannes Thumshirn wrote:

...

On Thu, Jun 02, 2016 at 04:21:22PM -0400, Greg Freemyer wrote:

...

On Thu, Jun 2, 2016 at 3:56 PM, Andrei Borzenkov wrote:

...

02.06.2016 16:27, Greg Freemyer пишет: === M.2 M Key device using AHCI protocol ===

Samsung XP941 SSD PCIe M Key 2280 MZHPU128HCGM-00000

AHCI is _not_ NVMe!!!

While that is true ...

...

AHCI is SATA.

... this is not always. M.2 storage can be connected either using SATA to HBA on motherboard or it can be connected using PCIe to PCIe switch on motherboard. In the latter case it can either contain AHCI compatible controller *on M.2 card itself* or it can contain NVMe compatibel controller. In both cases host interface is still PCIe which provides advantage over plain SATA.

The above card seems to be AHCI over PCIe, so no SATA :)

OK, this card above connects the Serial ATA AHCI via PCIe to your CPU instead of using the SoC's or Chipset's AHCI. [1] is a good read for that. Protocol and kernel driver wise (which I think is what is of interest here) AHCI will be handled by ahci.ko, libahci.ko, scsi_mod.ko and sd.ko. This will give you the beloved /dev/sd* devices. [2] has a nice diagram for the principal operation. Whereas real NVMe will be handled by nvme-core.ko and nvme.ko giving you /dev/nvme* devices. The handling of both is _very_ different. The biggest advantage of NVMe for instance would be true multi queue capability. From the hardware as well as the kernel's block layer and hw driver. This means, you can actually submit I/O from different CPU cores and it'll be traveling down the stack on these cores causing no lock contention as it never has to be touched by other cores. [1] http://www.intel.com/content/www/us/en/io/serial-ata/serial-ata-ahci-spec-re... [2] https://www.thomas-krenn.com/de/wikiDE/images/2/2d/Linux-storage-stack-diagr... But enough of that, I didn't intend to start a flame war here. Byte, Johannes -- Johannes Thumshirn Storage jthumshirn@suse.de +49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 2016-06-03 10:26, Andrei Borzenkov wrote:

On Fri, Jun 3, 2016 at 11:06 AM, Johannes Thumshirn wrote:

...

On Fri, Jun 03, 2016 at 10:48:05AM +0300, Andrei Borzenkov wrote:

...

On Fri, Jun 3, 2016 at 10:31 AM, Johannes Thumshirn wrote:

...

On Thu, Jun 02, 2016 at 04:21:22PM -0400, Greg Freemyer wrote:

...

On Thu, Jun 2, 2016 at 3:56 PM, Andrei Borzenkov wrote:

...

02.06.2016 16:27, Greg Freemyer пишет: === M.2 M Key device using AHCI protocol ===

Samsung XP941 SSD PCIe M Key 2280 MZHPU128HCGM-00000

AHCI is _not_ NVMe!!!

While that is true ...

...

AHCI is SATA.

... this is not always. M.2 storage can be connected either using SATA to HBA on motherboard or it can be connected using PCIe to PCIe switch on motherboard. In the latter case it can either contain AHCI compatible controller *on M.2 card itself* or it can contain NVMe compatibel controller. In both cases host interface is still PCIe which provides advantage over plain SATA.

The above card seems to be AHCI over PCIe, so no SATA :)

OK, this card above connects the Serial ATA AHCI via PCIe to your CPU instead of using the SoC's or Chipset's AHCI. [1] is a good read for that.

Do you say that this card internally has SATA link between actual storage and host controller? Do you *know* it? How can it achieve 1170 MB/s (from data sheet) during sequential read in this case?

Do not be confused by reusing AHCI as host controller interface; it does not imply that internal data transfer must be SATA.

...

Protocol and kernel driver wise (which I think is what is of interest here) AHCI will be handled by ahci.ko, libahci.ko, scsi_mod.ko and sd.ko. This will give you the beloved /dev/sd* devices. [2] has a nice diagram for the principal operation.

Yes. And that is exact reason for existence of AHCI emulation in NVMe devices - because they do not require additional drivers and are fully transparent. It does not really mean that NVMe devices are using SATA *internally*.

So will I have to report a bug ? nvme version nvme version 0.5 qt-kt:/ # nvme list /dev/nvme0n1 nvme0 failed to map qt-kt:/ # nvme list /dev/nvme nvme0 failed to map qt-kt:/ # lsmod | grep nvme nvme 73728 4 qt-kt:/ # nvme list-ctrl /dev/nvme0n1 NVMe Status:INVALID_FIELD(2002) cntid:0 qt-kt:/ # nvme list-ctrl /dev/nvme0 NVMe Status:INVALID_FIELD(2002) cntid:0 qt-kt:/ # nvme list-ctrl /dev/nvme0 NVMe Status:INVALID_FIELD(2002) cntid:0 qt-kt:/ # nvme smart-log /dev/nvme0n1 Smart Log for NVME device:nvme0n1 namespace-id:ffffffff critical_warning : 0 temperature : 33 C available_spare : 100% available_spare_threshold : 50% percentage_used : 0% data_units_read : 7915800 data_units_written : 39064599 host_read_commands : 88286900 host_write_commands : 1142492157 controller_busy_time : 1348 power_cycles : 555 power_on_hours : 2722 unsafe_shutdowns : 200 media_errors : 0 num_err_log_entries : 134 Warning Temperature Time : 0 Critical Composite Temperature Time : 0 Temperature Sensor 1 : 0 C Temperature Sensor 2 : 0 C Temperature Sensor 3 : 0 C Temperature Sensor 4 : 0 C Temperature Sensor 5 : 0 C Temperature Sensor 6 : 0 C Temperature Sensor 7 : 0 C Temperature Sensor 8 : 0 C qt-kt:/ # nvme error-log /dev/nvme0n1 Error Log Entries for device:nvme0n1 entries:64 ................. Entry[ 0] ................. error_count : 134 sqid : 0 cmdid : 0 status_field : 0x4004 parm_err_loc : 0 lba : 0 nsid : 0 vs : 0 ................. Entry[ 1] ................. error_count : 133 sqid : 0 cmdid : 0 status_field : 0x4004 parm_err_loc : 0 lba : 0 nsid : 0 vs : 0 ................. -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, Jun 03, 2016 at 11:26:22AM +0300, Andrei Borzenkov wrote:

On Fri, Jun 3, 2016 at 11:06 AM, Johannes Thumshirn wrote:

...

OK, this card above connects the Serial ATA AHCI via PCIe to your CPU instead of using the SoC's or Chipset's AHCI. [1] is a good read for that.

Do you say that this card internally has SATA link between actual storage and host controller? Do you *know* it? How can it achieve 1170 MB/s (from data sheet) during sequential read in this case?

Do not be confused by reusing AHCI as host controller interface; it does not imply that internal data transfer must be SATA.

OK, I think we're mixing up the physical and protocol layers here. I actually don't care what's the physical connection internally. For the kernel it is SATA with all it's up and downs.

...

Protocol and kernel driver wise (which I think is what is of interest here) AHCI will be handled by ahci.ko, libahci.ko, scsi_mod.ko and sd.ko. This will give you the beloved /dev/sd* devices. [2] has a nice diagram for the principal operation.

Yes. And that is exact reason for existence of AHCI emulation in NVMe devices - because they do not require additional drivers and are fully transparent. It does not really mean that NVMe devices are using SATA *internally*.

Correct. For the kernel this is just a plain old libahci/libata device. Which one one hand gives you the warm and cosy feeling of /dev/sd* and the scsi midlayer and all the libata (and sub libraries) goodness. But this also implies only one queue and a maximal queue depth of 32 (with NCQ). This might not be a problem at all for given the target platform is a laptop, but highly I/O intensive parallel workloads (i.e. a kernel compile) highly benefit from the true parallelism of a NVMe device. Take for example the device I've posted as a M.2 NVMe example. It implements 8 Hardware queues # nvme get-feature /dev/nvme0 -f 7 get-feature:0x07 (Number of Queues), Current value: 0x070007 and # ls /sys/block/nvme0n1/mq/ 0 1 2 3 4 5 6 7 These hardware queues now get mapped to per-cpu software queus: # cat /sys/block/nvme0n1/mq/*/cpu_list 0, 1, 12, 13 2, 14 3, 4, 15, 16 5, 17 6, 7, 18, 19 8, 20 9, 10, 21, 22 11, 23 Also no I/O scheduling of any kind is done on these devices. So if Greg really want's to go down the road and by one of these, I'd go for a true NVMe one. Let me quote [1] Section 3 NVMe and AHCI Comparison: "While SATA Express/AHCI has the benefit of legacy software compatibility, the AHCI interface does not deliver optimal performance when talking to a PCIe SSD. This is because AHCI was developed at a time when the purpose of the HBA in a system was to connect the CPU/Memory subsystem with the much slower rotating media-based storage subsystem. Such an interface has some inherent inefficiency when applied to SSD devices, which behave much more like DRAM than spinning media. NVMe has been designed from the ground up to exploit the low latency of today’s PCIe- based SSD’s, and the parallelism of today’s CPU’s, platforms, and applications. [...]"

...

...

But enough of that, I didn't intend to start a flame war here.

I do not consider it flame war at all. Technology (both NVMe and M.2) is relatively new, there are many marketing buzzwords but not as much technical facts. So it is good if we all can get better understanding.

Good to hear. Marketing buzzword wise, these cards are called SATA Express IIRC. [1] https://www.sata-io.org/sites/default/files/documents/NVMe%20and%20AHCI%20as... Byte, Johannes -- Johannes Thumshirn Storage jthumshirn@suse.de +49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Jun 2, 2016 at 8:42 AM, Andrei Borzenkov wrote:

On Thu, Jun 2, 2016 at 12:38 PM, Greg Freemyer wrote:

...

Hoping to get access to a laptop with a NVMe / M.2 storage subsystem later today. (a Alienware laptop purchased in March or April).

Is it NVMe pure, without any other mass media? This should be challenging for bootloader configuration. Looking forward for your bug reports :)

Andrei, I've taken a Asus x99 Deluxe MB based computer and disconnected all SATA devices. I put a NVMe SSD into the MB. (It has a M.2 M-Key slot). Leap 42.2 Alpha 1 installed with no problems. It is up and running. (I am having some video issue in KDE, but that isn't relevant to NVMe.) Are there any tests you'd like me to run? FYI: I will likely upgrade to Leap 42.2 alpha 1 early next week. I actually installed from the wrong DVD, didn't mean to use alpha 1 at all. Greg -- Greg Freemyer www.IntelligentAvatar.net -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org