Continuous conversion

[superware]

Hello,

I'm successfully running the convert example, but how should libpisp be used for continuous conversion? How to queue the buffers after consumption?

void PiSPConverter::transform(GstBuffer *inbuf)
{
    GstMapInfo map;

    if (!gst_buffer_map(inbuf, &map, GST_MAP_READ)) {

		std::cerr << "gst_buffer_map failed" << std::endl;
		exit(-1);
    }

    auto buffers = _backend_device->GetBuffers();
    auto buffer = buffers.find("pispbe-input");

    memcpy(buffer->second.mem[0], map.data, buffer->second.size[0]);

    int ret = _backend_device->Run();
	if (ret)
	{
		std::cerr << "Job run error!" << std::endl;
		exit(-1);
	}

    gst_buffer_unmap(inbuf, &map);

    buffer = buffers.find("pispbe-output0");

    memcpy(_params.output_data, buffer->second.mem[0], buffer->second.size[0]);
}

Unable to queue buffer: Invalid argument

Thanks.

[naushir]

I assume you are not running with the top-of-tree libpisp for your application? BackendDevice::GetBuffers() has been replaced with BackendDevice::AcquireBuffers(), and once you have dequeued the buffers after a job, I think you ought to just be able to just use the same buffer set when calling Run again. Can you give that a try?

[superware]

Hello @naushir, I'm on 1.2.1, when is the next release due? Is it not possible to enqueue the buffers back in the current version?

[naushir]

Hello @naushir, I'm on 1.2.1, when is the next release due?

There is no immediate plans for a package release unfortunately. If you are able to, I suggest building/installing libpisp manually to test with your application.

Is it not possible to enqueue the buffers back in the current version?

If I recall correctly, I think the old code only handled one-shot operations. So this newer code is likely to be needed for re-queuing type operations.

[superware]

Hi @naushir, ok so after building top-of-tree, and in my_element:

void V4l2PiSPConverter::initialize(TransformParams& params)
{
    _params = params;
    _devices = new libpisp::helpers::MediaDevice();
    _media_dev = _devices->Acquire();

    if (_media_dev.empty()) {
        std::cerr << "Unable to acquire any pisp_be device!" << std::endl;
        exit(-1);
    }

    _backend_device = new libpisp::helpers::BackendDevice(_media_dev);

    const std::vector<libpisp::PiSPVariant> &variants = libpisp::get_variants();
    const media_device_info info = _devices->DeviceInfo(_media_dev);
    auto variant = std::find_if(variants.begin(), variants.end(),
                                [&info](const auto &v) { return v.BackEndVersion() == info.hw_revision; });
    if (variant == variants.end())
    {
        std::cerr << "Backend hardware could not be identified: " << info.hw_revision << std::endl;
        exit(-1);
    }

    _variant = variant;

    _be = new libpisp::BackEnd(libpisp::BackEnd::Config({}), *_variant);
    _be->GetGlobal(_global);
    _global.bayer_enables = 0;
    _global.rgb_enables = PISP_BE_RGB_ENABLE_INPUT + PISP_BE_RGB_ENABLE_OUTPUT0;

    pisp_image_format_config i = {};
    i.width = params.input_width;
    i.height = params.input_height;
    i.stride = params.input_stride;
    i.stride2 = params.input_stride2;
    gst_to_pisp_format(params.input_format, i);
    assert(i.format);
    _be->SetInputFormat(i);

    pisp_be_output_format_config o = {};
    o.image.width = params.output_width;
    o.image.height = params.output_height;
    gst_to_pisp_format(params.output_format, o.image);
    assert(o.image.format);
    libpisp::compute_optimal_stride(o.image, true);
    _be->SetOutputFormat(0, o);

    params.output_stride = o.image.stride;
    params.output_stride2 = o.image.stride2;

    _be->SetGlobal(_global);
    _be->SetCrop(0, { 0, 0, i.width, i.height });
    _be->SetSmartResize(0, { o.image.width, o.image.height });

    _be->Prepare(&_tiles_config);

    _backend_device->Setup(_tiles_config);
    _buffers = _backend_device->AcquireBuffers();

    std::cerr << "!!" << std::endl;
}

void V4l2PiSPConverter::transform(GstBuffer *inbuf)
{
    std::cerr << "!!!" << std::endl;

    GstMapInfo map;

    if (!gst_buffer_map(inbuf, &map, GST_MAP_READ)) {

        std::cerr << "gst_buffer_map failed" << std::endl;
        exit(-1);
    }

    auto buffer = _buffers.find("pispbe-input");

    memcpy(buffer->second.mem[0], map.data, buffer->second.size[0]);

    int ret = _backend_device->Run(_buffers);
    if (ret)
    {
        std::cerr << "Job run error!" << std::endl;
        exit(-1);
    }

    gst_buffer_unmap(inbuf, &map);

    buffer = _buffers.find("pispbe-output0");

    memcpy(_params.output_data, buffer->second.mem[0], buffer->second.size[0]);
}

Running:

$ gst-launch-1.0 libcamerasrc ! video/x-raw,format=BGR,width=1280,height=800,framerate=30/1 ! my_element ! fakesink
...
[10017]  INFO Camera camera.cpp:1215 configuring streams: (0) 1280x800-RGB888/Rec709/Rec709/None/Full
[10015]  INFO RPI pisp.cpp:1483 Sensor: /base/axi/pcie@1000120000/rp1/i2c@80000/ov9281@60 - Selected sensor format: 1280x800-Y10_1X10/RAW - Selected CFE format: 1280x800-PC1M/RAW/Linear/Rec601/Limited
!!
[10015] ERROR V4L2 v4l2_videodevice.cpp:1322 /dev/video20[18:out]: Unable to request 32 buffers: Device or resource busy
[10015] ERROR RPI pipeline_base.cpp:673 Failed to allocate buffers
0:00:00.154558472 10013 0x555608da7000 WARN            libcamerasrc gstlibcamerasrc.cpp:871:gst_libcamera_src_task_enter:<libcamerasrc0> error: Failed to start the camera: Device or resource busy
0:00:00.154586176 10013 0x555608da7000 WARN            libcamerasrc gstlibcamerasrc.cpp:871:gst_libcamera_src_task_enter:<libcamerasrc0> error: Camera.start() failed with error code -16
ERROR: from element /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Failed to start the camera: Device or resource busy
Additional debug info:
../src/gstreamer/gstlibcamerasrc.cpp(871): gst_libcamera_src_task_enter (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:
Camera.start() failed with error code -16
Execution ended after 0:00:00.017029717
Setting pipeline to NULL ...
Freeing pipeline ...

So V4l2PiSPConverter::initialize executes (on my_element set_caps), but then libcamerasrc fails before processing any frame. How come initialize() is affecting libcamerasrc? Aren't those two different contexts?

Thanks!

[naushir]

I think what's happening is contention of the V4L2 backend media device that is used by both the libpisp converter and libcamerasrc. There's currently no easy way to share the V4L2 media device between different consumers.

One suggestion would be to hardcode the media device you use in _media_dev = _devices->Acquire();. This function takes an optional string parameter for the media device to use. You can list the media devices with _devices->List(). Try always selecting the second device and see if that removes the contention?

[superware]

Thanks @naushir, I'll have to build top-of-tree again and give it a try, will report back.

Can Run actually receive any externally allocated DMA-BUF buffers (any v4l2_buffer wrapped with V4l2Device::Buffer) to maintain zero-copy pipeline?

---

EDIT: I'm trying to come up with a custom v4l2convert that uses PiSP for fast format conversion, cropping and resizing while allowing zero-copy (currently with v4l2src but later with libcamerasrc), for example:

v4l2src io-mode=dmabuf ! video/x-raw,format=RGB,width=1920,height=1080,framerate=30/1 ! dmabufelement1 ! dmabufelement2 ! v4l2convert ! video/x-raw,format=RGBA ! dmabufopengl3 ! v4l2convert ! video/x-raw,width=1280,height=720 ! x264enc ! ...

The input DMA-BUF buffers are allocated by v4l2src, v4l2convert creates an internal DMA-BUF allocator and allocates the output buffers. What I've been trying for the past day, without much success, is setting up libpisp to process these existing buffers. pispbe-input and pispbe-output0 are requested and defined with V4L2_MEMORY_DMABUF and pispbe-config remains as V4L2_MEMORY_MMAP (using custom V4l2DeviceDma and MediaDeviceDma). pispbe-input and pispbe-output0 are being streamed and queued, but _nodes.at("pispbe-config").QueueBuffer(_config_buffer.buffer.index) fails with "Unable to queue buffer: Invalid argument".

So a) Does PiSP / libpisp architecture allow parallel processing of multiple jobs/contexts? b) Is there any support for DMA-BUF allowing high performance zero-copy processing? Is this a hardware/software limitation?

Thank you for helping out.

[naushir]

Can Run actually receive any externally allocated DMA-BUF buffers (any v4l2_buffer wrapped with V4l2Device::Buffer) to maintain zero-copy pipeline?

Not yet I don't think. But it might be easy enough to add an "import" buffer function I think.

So a) Does PiSP / libpisp architecture allow parallel processing of multiple jobs/contexts? b) Is there any support for DMA-BUF allowing high performance zero-copy processing? Is this a hardware/software limitation?

Yes it does. But how we achieve this (in the absence of a supported V4L2 kernel API for resource sharing) is to have duplicate media device nodes for the same hardware block. The kernel driver marshals the use of the hardware between these multiple userland devices. However, the key is that the muliple userland users cannot use the same media device node.

[superware]

Can Run actually receive any externally allocated DMA-BUF buffers (any v4l2_buffer wrapped with V4l2Device::Buffer) to maintain zero-copy pipeline?

Not yet I don't think. But it might be easy enough to add an "import" buffer function I think.

That will be great. From what I see (searching GitHub for example), the direct usage in libpisp is extremely scarce although it's a super important RPi infrastructure, so improving support for "real life" scenarios along with good documentation will probably lead to exponential adoption rates. In the meantime, should the zero-copy setup I'm trying which defines V4L2_MEMORY_DMABUF on pispbe-input and pispbe-output0 (and of course FD per plane etc.) while leaving pispbe-config as-is, work? It seems that nothing fails up until pispbe-config queueing ("Invalid argument").

So a) Does PiSP / libpisp architecture allow parallel processing of multiple jobs/contexts? b) Is there any support for DMA-BUF allowing high performance zero-copy processing? Is this a hardware/software limitation?

Yes it does. But how we achieve this (in the absence of a supported V4L2 kernel API for resource sharing) is to have duplicate media device nodes for the same hardware block. The kernel driver marshals the use of the hardware between these multiple userland devices. However, the key is that the muliple userland users cannot use the same media device node.

By "duplicate media device nodes" you mean there's only a specific hardcoded set of duplicates, or that it's possible to add more duplicates manually by tweaking media-ctl for example. Just a thought, why not use a system-wide mutex in libpisp to indicate device node usage? It doesn't make much sense to bring this kind of resource distribution to the userland.

[naushir]

In the meantime, should the zero-copy setup I'm trying which defines V4L2_MEMORY_DMABUF on pispbe-input and pispbe-output0 (and of course FD per plane etc.) while leaving pispbe-config as-is, work? It seems that nothing fails up until pispbe-config queueing ("Invalid argument").

Can you check the dmesg logs to see if the kernel driver tells why the config queue action fails?

By "duplicate media device nodes" you mean there's only a specific hardcoded set of duplicates, or that it's possible to add more duplicates manually by tweaking media-ctl for example.

The kernel driver exposes multiple sets of device nodes, one per-user of the device, see:
https://github.com/raspberrypi/linux/blob/99972b2fa5395542e7c24e4d894be5ede383055f/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c#L38

Just a thought, why not use a system-wide mutex in libpisp to indicate device node usage? It doesn't make much sense to bring this kind of resource distribution to the userland.

That assumes all users of the ISP (libcamera, libpisp, others...) will know about this mutex, which cannot be guaranteed. There is work to adapt V4L2 to all multiple userland clients of a single media device, but it's not laded yet. When it does, this problem ought to go away.

[superware]

In the meantime, should the zero-copy setup I'm trying which defines V4L2_MEMORY_DMABUF on pispbe-input and pispbe-output0 (and of course FD per plane etc.) while leaving pispbe-config as-is, work? It seems that nothing fails up until pispbe-config queueing ("Invalid argument").

Can you check the dmesg logs to see if the kernel driver tells why the config queue action fails?

Nothing in dmesg. Turns out the 1st job is (kind of) successful, only the 2nd VIDIOC_QBUF fails:

1. Transforming 1280x720xRGB24 to 640x360xRGB24 - works only for the 1st job, output is good.
2. Transforming 1280x720xRGB24 to 640x360xRGBX32 - works only for the 1st job, output has some wide vertical black gaps.
3. Compared structs between jobs, everything seems to be the same except for the DMA-BUF FDs which are different in the 2nd job (obviously GStreamer allocated a few).

Any ideas?

By "duplicate media device nodes" you mean there's only a specific hardcoded set of duplicates, or that it's possible to add more duplicates manually by tweaking media-ctl for example.

The kernel driver exposes multiple sets of device nodes, one per-user of the device, see: https://github.com/raspberrypi/linux/blob/99972b2fa5395542e7c24e4d894be5ede383055f/drivers/media/platform/raspberrypi/pisp_be/pisp_be.c#L38

So only two sets, for example one libcamerasrc and one custom v4l2convert, which is quite limiting.

Just a thought, why not use a system-wide mutex in libpisp to indicate device node usage? It doesn't make much sense to bring this kind of resource distribution to the userland.

That assumes all users of the ISP (libcamera, libpisp, others...) will know about this mutex, which cannot be guaranteed. There is work to adapt V4L2 to all multiple userland clients of a single media device, but it's not laded yet. When it does, this problem ought to go away.

Do you mean an M2M device like RPi4 /dev/video12?

[naushir]

Nothing in dmesg. Turns out the 1st job is (kind of) successful, only the 2nd VIDIOC_QBUF fails:

1. Transforming 1280x720xRGB24 to 640x360xRGB24 - works only for the 1st job, output is good.

2. Transforming 1280x720xRGB24 to 640x360xRGBX32 - works only for the 1st job, output has some wide vertical black gaps.

3. Compared structs between jobs, everything seems to be the same except for the DMA-BUF FDs which are different in the 2nd job (obviously GStreamer allocated a few).

Any ideas?

I presume you are re-configuring and reallocating buffers between steps 1 and 2?

Do you mean an M2M device like RPi4 /dev/video12?

The RPi 4 device driver will suffer the same issue of multiple users.

[6by9]

Do you mean an M2M device like RPi4 /dev/video12?

The RPi 4 device driver will suffer the same issue of multiple users.

A driver that uses the V4L2 M2M framework and has one input and one output stream will give a new context for every file handle - it's documented that way at https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-mem2mem.html
The V4L2 stateful decoder (/dev/video10), encoder (/dev/video11), and resize (/dev/video12) on Pi4 all fall into that category.

When there is more than one input and one output stream then they have to be separated across multiple /dev/video devices, and the assumption made that only one application will open all those nodes at the same time. That is the case with the full ISP on Pi0-4 (/dev/video13 to /dev/video16) and PiSP backend on Pi5.
The only way to handle multiple contexts in this case is to create a full duplicate set of device nodes for additional contexts, hence /dev/video20 - /dev/video23 on Pi4, and similar on Pi5 (I can't check the numbers quickly)

[superware]

@6by9 Yes, on RPi4 I was able to use /dev/video12 "at will", which was great, much like NVidia's NvBufSurfTransform.

There is work to adapt V4L2 to all multiple userland clients of a single media device, but it's not laded yet. When it does, this problem ought to go away.

@naushir Sounds very promising.

Regarding my RGB888 to RGBX8888 issue - I missed this special case handling in convert.cpp on main:

// Hack to generate RGBX even when BE_MINOR_VERSION < 1 using Resample

So now this path seems to work well. The only issue left is the original one - using libpisp continuously, which fails on the 2nd job. Currently I have to reconfigure and reallocate everything per frame, which doesn't make much sense at all.

Thank you.

[naushir]

You will have to reconfigure and reallocate if you change formats, crops and resolutions. Buffer sizes change and so do the tiling parameters, this is unavoidable.

[superware]

Wasn't trying to change format, crops or resolution between jobs, just process continuesly. Nothing fails up until the 2nd ispbe-config queueing ("Invalid argument").