Streamer/Tape support #93
Comments
|
@bog-dan-ro Thank you for creating this ticket. Currently there are no plans to add streamer support to SCSI2Pi. In the past tapes were used for backups, because there was no better/cheaper solution. Nowadays, especially with tools like SCSI2Pi, you simply use a drive image for a backup. This is faster and more flexible than a streamer emulation. Using a streamer emulation instead would be rather inconvenient. However, as soon as you have added fully functional tape support to PiSCSI please get back to me. I wil then run tests with my Atari (I have streamer software for it and also an HP streamer), and I will then re-evaluate this ticket based on the results of my tests. |
|
@bog-dan-ro By the way, I would be interested to learn which methods from the Disk class you think should be moved into the StorageDevice class. |
|
Actually they are not too many, check my last commit from https://github.com/bog-dan-ro/piscsi |
|
See bog-dan-ro/piscsi@52a50c9#diff-ba201351e19eac2f5e902b9308a59f1e3bdc19ed324119dab011d8336c345410 I didn't moved the caching as streamers do not need it as they are sequential so, once you read/write a block it will not seek back to read it again. |
|
I will have a look at the code you moved and will get back to you, but this may take some time. Is the streamer code in your repository already tested and working? |
There are other use-casese for than tape than backup. For instance, Altos 386 Series 1000 needs it to install the the system utilities.
No rush, I'll be in vacation for a couple of weeks.
I tested it a bit, but sadly is not working ... The mandatory implementation is there, but I don't know how to properly debug it. Most probably I'll have to add a modern build system (cmake?) so my IDE to know how to cross compile, deploy & debug it properly. I chose tandber as the reference device for two reasons:
|
|
Indeed, I missed the use case of an installation. Anyway, I will get back to you (presumably in a couple of days) regarding moving code from Disk to StorageDevice. There is no direct relationship between cmake and cross-compling, by the way. The nightly SCSI2Pi builds and the release builds are cross-compiled, but SCSI2Pi does not use cmake. I thought about using it, but it would not be worth the effort, considering that the Makefile is portable and straightforward. |
Nope is not, at least not direct. Current makefile works great to cross-compile it from command line. I created a docker file to help me cross compile it, and I have no issues to cross-compile it.
No hurry at all, starting next Friday I'll be in vacation for two weeks. |
|
@bog-dan-ro In the develop branch I moved the code related to sector/block sizes from the Disk class to the StorageDevice class. |
|
@uweseimet I'll check it once I'm back. |
|
@uweseimet Here PiSCSI/piscsi#1477 is the PR |
|
@bog-dan-ro Can you please remove my name in the copyright notice of the new streamer class code (.cpp and .h)? |
Done |
|
@bog-dan-ro In the meantime I have added a branch issue_93 with initial tape drive support to SCSI2Pi. This implementation is completely new, i.e. it is not based on any existing code. I have already successfully tested this branch with tar, both in the tar-file compatible mode (file extension .tar), and in the full-featured mode with filemarks support. When writing files to a real device instead of the filesystem tar actually makes use of filemarks. |
|
@bog-dan-ro There has not been any recent feedback, neither in this ticket nor in PiSCSI/piscsi#480. As far as the SCSI2Pi ticket is concerned is is essentially done and tested. What about your PiSCSI ticket/PR? @Pacjunk @sidick @ppuskari Quite long ago you were interested in tape support for the PiSCSI board. In case you are still interested, I would appreciate help with testing. |
|
This is wonderful news. I am about ready to test this after building. I have been in the middle of a major job change so have been out of pocket a bit.
Looking forward to seeing if the IIgs software can properly work with the new code and recognize it as the 3m drive and 40meg block formatted and sellable capacity. If this works I am going to learn a lot from your code and I am very appreciative!
Petar Puskarich
***@***.***
…Sent from my iPhone
On Oct 5, 2024, at 5:01 AM, Uwe Seimet ***@***.***> wrote:
@bog-dan-ro<https://github.com/bog-dan-ro> There has not been any recent feedback, neither in this ticket nor in PiSCSI/piscsi#480<PiSCSI/piscsi#480>. As far as the SCSI2Pi ticket is concerned is is essentially done and tested. What about your PiSCSI ticket/PR?
I would like to ensure that both implementations are compatible from the user's perspective. For SCSI2Pi there are binaries of the development branch with all changes included, see https://www.scsi2pi.net/packages/develop/.
See the comments in the header of tape.cpp for details on the SCSI2Pi implementation.
@Pacjunk<https://github.com/Pacjunk> @sidick<https://github.com/sidick> @ppuskari<https://github.com/ppuskari> Quite long ago you were interested in tape support for the PiSCSI board. In case you are still interested, I would appreciate help with testing.
—
Reply to this email directly, view it on GitHub<#93 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIMVUB5TTRBYJCTSTB5FILZZ62GTAVCNFSM6AAAAABMWBHWR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGAYDGMZXHA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Hi @uweseimet, Sorry for slow reply, I was quite busy at work. BogDan. |
|
@bog-dan-ro Backporting is fine, but what about the existing PiSCSI PR? As far as I can tell the PiSCSI team has been waiting for your feedback for weeks. |
|
@uweseimet my patch works for me and from my POV it's done. There are a few small issues to fix on the UI side which I could not fix them myself. |
|
When will these issues be fixed? What about the device type name? Please see my last comment in the PiSCSI ticket. |
Hi Uwe - I've been out of the loop for a fair while but I am attempting to have a look at your new code. Taking me a while to get my head around doing things with a command line! I updated my system to the latest PiSCSI (3.5 hours!) and had it working. I then installed your package. It disabled PiSCSI (expected), but did not create a system service for scsi2pi - I did this. The web interface does not seem to work now, even though some of documentation says it will. I added a disk and a few .TAP files to the configuration. The workstation can see the vendor details for the disk and tape drives at the console prompt - so a good start. First try booting nothing worked, second try failed during boot with all devices being detached part way through the boot. Third try and I booted the VAXstation off the disk image. When I try and mount a tape image (which I have verified as being correct under SimH) the system just says that the device is offline. So looks like a bit more debugging is required. So my questions are: 1 - I can't find the log file anywhere. Where is it, and what is the best way to change the logging level to trace Thanks, |
|
@Pacjunk Thank you for your feedback! Regarding the system service, I may have missed updating the installation webpage. Initially the PiSCSI service was shut down and the SCSi2Pi service was started up instead, when running "apt install". I removed the last step, but I have to admit I don't know why anymore ;-). I think it was because it is sometimes annoying to have a service not just created but also automatically started without explicitly configuring this. I just updated the "Quickstart" section of the website. Regarding your questions:
s2p is usually configured with a property file. http://www.scsi2pi/en/properties.html. The command line options complement/override property file settings. Some configuration data, especially mode pages, require a property file. Add all available options to a sample configuration file would probably make this file quite confusing. It would also mean maintenance overhead, because with each change I would not only have to update the help texts, the manpages and the website, but also this file. As already mentioned, for testing purposes using the command line instead of the system service is usually more convenient, as you can quickly stop s2p, change the parameters, and restart.
I have tested the current develop branch with tar and mtx, but when using other (or more elaborate) tape tools it may turn out that the implementation has to be updated/changed. I think that it is compliant with the standard, but some tools may need some of the optional commands not implemented. Or bugs still have to be fixed that tar does not trigger. I hope this helps to get you started. |
|
Here is a trace of my unsuccessful attempt to mount a tape on my VAXstation. Don't know how useful this is, but anyway... |
|
@Pacjunk Thank you. Looks as if the problem is related to an operation mode that is not implemented by SCSI2Pi: This command results in an error because SCSI2Pi does not support non-fixed tape blocks. But your workstation also sends a MODE SELECT to configure something. |
|
@Pacjunk As far as I can tell the MODE SELECT tries to configure the density, setting it to the default value 0. 0 means that the device selects its default density, which is fine for SCSI2Pi, because other densities are not supported (or faked, because there is no density) anyway. The fact that your workstation eventually tries to read data indicates that it may be fine with whatever happened before (MODE SELECT/MODE SENSE). The first actual error occurs when your machine tries to read a block of 80 bytes, which is currently not supported by SCSI2Pi. Supporting non-fixed block sizes is optional (and not required by tar). I will add this feature most likely tomorrow. Without this feature I don't expect a different behavior when configuring SCSI2Pi with a different blocksize ("-b 1024"), but I'd still like to ask you for doing this. It may provide further information. Please only use the debug log level, in order to keep the logfile small. Do you have a real streamer that works correctly with your workstation? In this case you can run some SCSI commands against it (using s2pexec, the SCSI2Pi Wiki provides details about this tool). Knowing the mode pages data of your streamer might help to improve the emulation. Which Pi model and which OS are you using for your tests? |
|
@Pacjunk There is an issue_93_2 branch now, which is supposed to support variable block sizes between 1 and 4096 bytes. Can you please test whether something changes with this branch? But before doing that can you please test with the develop branch and a block size of 1024 bytes and comment in my questions in my previous comment? Thank you! I cannot test flexible block sizes with the software I have, but I verified that tar still works with the issue_93_2 branch, and that tapeinfo reports the expected minimum and maximum sizes: |
|
Thanks for that. I'll be away for a few days, so can't test until next week. I assume the first thing that the workstation does is look for the tape label. Yes I have some real tape drives - hopefully they and the tapes still work! I'm pretty sure that I dont get the device offline message with a blank tape - probably will just say label not found or not initialised. I'll have to try and bulk erase a tape and give it a try. I'm using a pi zero W with bullseye. How do I get a copy of that branch? Is there a .deb file for it, or do I have to do a full build? |
|
@Pacjunk I have just triggered building .deb binaries for commit ID ee8ba06 in the issue_93_2 branch. The development binaries have the commit ID in their filename. It would be very helpful if you could collect information on your real drives. You can do this by connecting your streamers to a SCSI chain with the PiSCSI board in it and running some SCSI commands. Just connect a streamer and then run commands with s2pexec, e.g.: This sequence of commands runs INQUIRY, gets all mode pages with MODE SENSE and separately retrieves page 0. With these data we will know more about the features of your drives. Regarding s2pexec also see https://github.com/uweseimet/scsi2pi/wiki/Advanced-Testing. |
|
@uweseimet
Overall your code has looks better than mine, it has also proper unit tests! [0] I temporary removed it and sent it to a friend to try to reverse engineering it. This computer uses custom RAM modules. |
|
@bog-dan-ro You have probably missed that code for MODE SENSE/SELECT, LOAD/UNLOAD and mode page handling is inherited from the StorageDevice class. |
|
@Pacjunk The issue_93_2 branch has been merged into the develop branch. |
|
Tried with block size set to 1024, got errors to do with no label (it does have a label). Tried with your latest code (as of a couple of days ago), and get device errors. Debug log attached... There is a SCSI test utility that tests various mode pages. I will run this and get back to you... |
|
Here is the output from the SCSI test utility. I haven't used the scsi2pi tester yet as I need to either find some different cables or an external tape drive that works (I'm sure there is one somewhere!). First file is with the SCSI2Pi running... Next is connected to a Sony DAT drive. This is a generic drive and doesn't have any DEC specific functions. and finally, this is the command run again with a tape mounted. The information returned seems to be different. Surprised the drive and especially the tape worked. Must be 20+ years old. |
|
@Pacjunk Thank you for running these tests. There are some differences in the mode pages which are caused by SCSI2Pi supporting some pages the tape drive does not support. This is when " CHECK CONDITION seen" is reported for the tape drive, but no errors are reported for SCSI2Pi. These differences should not matter. In the SCSI2Pi log, is the " READ(6)/GET MESSAGE(6) ($08)" line the last line of the logfile, i.e. nothing happened after that? There are no SCSI errors logged anymore, but the sequence of MODE SELECT/MODE SENSE commands looks as if the workstation is trying to configure something. Can you please run the same test again, but with a block size of 4096 bytes and on trace level? Please use the very latest development binaries for everything.
It does have a label? I'm afraid I don't know what you mean, assuming that a label is an object on a tape. A blank image file does not have any data on it, i.e. it would also have no label. When you provide a logfile please also provide the commit ID of the binaries this log is based on, so that I can identify the codebase. |
|
@Pacjunk After running tests and collecting logs with the latest development binaries as outlined in my previous comment, I would like to ask you to run tests with the issue_93_3 branch. This branch you will have to compile yourself, and depending on future test results I will try to update it where required. In order to save compilation time I suggest using "make s2p ENABLE_SCHD=1 ENABLE_SCTP=1", so that only the s2p binary is created and only SCHD and SCTP are supported. I expect that future updates will only affect the tape.cpp file, i.e. compiling updates will not take that long even on a small Pi. In the issue_93_3 branch I modified the tape properties based on the mode page output of your tool, the mode pages of my HP streamer and information, and the SSC-5 specification, which is a bit more precise than the SCSI-2 specification. Nevertheless, there are some mode page fields/device properties where I do not know how to correctly support them, or where your workstation might expect certain not mandatory properties. I hope that a SCSI2Pi trace log of the issue_93_3 branch provides more insight. |
|
After applying a work-around (not related to SCSI2Pi) I can now successfully use the Gemar tape backup software for the Atari. This software uses a fixed block size and works similar to tar. |
|
@Pacjunk Any news? I'm afraid that without feedback on my previous questions and more testing from your side it will not be possible to support more software/platforms than those that are currently supported: mt, mtx, tar and the Gemar streamer software for the Atari. |
|
Sorry, life got in the way in a stressful and busy way. I will try to look soon. I assume I can just use the latest dev daily build? |
|
Yes, the latest daily build plus a self-compiled issue_93_3 branch, all with a block size of 4096 bytes, with log level trace. Also please see my previous comments. |
|
Took forever to build the issue_93_3 branch, but got there in the end. Attached is the log. Note there is some access to device 0 at the beginning. This is just the OS scanning the SCSI buss. Still getting a device error when trying to mount a tape. |
|
@Pacjunk Thank you for the log. Please (in further logs) suppress logging for device 0: by using "-L trace:3". Also please remember to provide trace logs for the current develop branch, and to add the property data of your other streamer, when you find the time for doing so. |
|
I used the latest develop branch, then applied 93_3 over the top - as you told me to do. THere is not much logging for 0 as I was not using it (booted off real HDD) |
|
I'm afraid there has been a misunderstanding. I would need logs on trace level for both branches. issue_93_3 differs from the develop branch. For the develop branch there are nightly binaries you can use. Please use the latest ones and provide the commit ID you use for testing. |
|
The last log does not reveal any SCSI errors. If your computer shows an error it is somehow not receiving the data it expects. I would still need the log for the develop branch, for comparison. I guess it will look very similar. |
|
@Pacjunk After revisiting your latest log I wonder whether you were really testing the issue_93_3 branch. The last read command in the log, when executed for a blank image or an image with random data, should report either MEDIUM ERROR or BLANK CHECK. The code says so, and I have verified this behavior by sendin: The error should be the same with the develop and the issue_93_3 branch. Old develop branch versions would not have reported an error but just returned the requested (empty or random) data. In order to avoid any potential confusion and mismatches between the code I am working on and the code you are testing, please:
Thank you! I would be interested in the response of your real tape drive when receiving the READ command above. Can you please add both the PiSCSI board and your drive to the same SCSI chain and then run s2pexec (e.g. with /opt/scsi2pi/bin/s2pexec) and then enter this: The first command will rewind the type. Please wait for the end of this operation (the tape drive may take some time for that) before running the second command, which is the same READ command as the one at the end of the logfile. And yet one more thing: Your logfile says "caching_mode=write-through". The caching mode is not a global property, but is device-specific (see https://www.scsi2pi.net/en/properties.html). The device specifier is missing, witout it the property is discarded. |
|
@Pacjunk You mentioned issues with the PiSCSI web UI and SCSI2Pi. While waiting for your feedback I verified whether the web UI is working with SCSI2Pi and did not notice any problems. I installed |
|
Hi, got a couple of hours today to do some testing using 13d6667 (although same results with 5cb8b4f). Using a blank tap file I managed to initialise (ie write a label), then mount the tape with that label successfully. Trying to copy files to the image resulted in errors. I notice that every record written to the .tap file has a "SCTP" prefix. This is most likely the reason that I cannot mount .tap files produced by others from places such as archive.org. Of course they don't have records prefixed with SCTP! I have attached two files. The .tap file (after it has been initialised) and the log file. The steps were: I probably won't get any time with this until next week as I will be away for a few days. Cheers, |
|
@Pacjunk Thank you for the log. Yes, the non-tar format is proprietary because it also contains meta data (see the comment at the beginning of tape.cpp) that represent filemarks and end-of-data markers. Without these meta data spacing in order to search for a filemark, for example, would not be possible. This is why spacing cannot be supported with tar files: The required markers are missing in these files. If I understand your correctly the log does not contain the SCSI operations executed when labeling a tape? Edit: But the logs now reveal more information. Looks as if the labeling step actually IS included, but please confirm. I see that your workstation/driver uses reverse spacing: This is not supported by SCSI2Pi (and also not a mandatory SCSI requirement) and I don't know if I will support it. See the initial comment of tape.cpp for details. |
|
Yes the log includes labelling the tape. |
Same request as PiSCSI/piscsi#480
I already have a non working WIP for piscsi (that was before I found scsi2pi).
I'll be happy to port it to scsi2pi. But I'll someones help, who knows the internals of this project, to fix the remaining stuff.
The text was updated successfully, but these errors were encountered: