Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# This workflow uses actions that are not certified by GitHub.
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
Expand Down Expand Up @@ -51,3 +51,18 @@ jobs:
with:
sarif_file: rust-clippy-results.sarif
wait-for-processing: true

- name: Install required cargo
run: cargo install cargo-tarpaulin

- name: Generate code coverage
run:
cargo tarpaulin
--all-features
--workspace --timeout 120 --out xml

- name: Upload to codecov.io
uses: codecov/codecov-action@v5
with:
token: ${{secrets.CODECOV_TOKEN}}
fail_ci_if_error: true
6 changes: 5 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed

- Compatible with meaningless input: Merging 0 or 1 dimensions will not change the layout;

### Added

- Add `to_inline_size` function, to copy data from `ArrayLayout<N>` into `ArrayLayout<M>`.
- Add `to_inline_size` function, to copy data from `ArrayLayout<N>` into `ArrayLayout<M>`;

## [0.2.1] - 2025-03-28

Expand Down
5 changes: 4 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,10 @@ name = "ndarray-layout"
description = "This crate provides definitions and transformations for multi-dimensional array data layouts."
version = "0.2.1"
edition = "2024"
authors = ["YdrMaster <ydrml@hotmail.com>"]
authors = [
"YdrMaster <ydrml@hotmail.com>",
"Simon25772 <ssh.king@qq.com>",
]
repository = "https://github.com/InfiniTensor/ndarray-layout.git"
documentation = "https://docs.rs/ndarray-layout"
license = "MIT"
Expand Down
42 changes: 41 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
[![Latest version](https://img.shields.io/crates/v/ndarray-layout.svg)](https://crates.io/crates/ndarray-layout)
[![Documentation](https://docs.rs/ndarray-layout/badge.svg)](https://docs.rs/ndarray-layout)
[![license](https://img.shields.io/github/license/InfiniTensor/ndarray-layout)](https://mit-license.org/)
[![codecov](https://codecov.io/github/Simon25772/ndarray-layout/branch/ShenghuSu/graph/badge.svg)](https://codecov.io/github/Simon25772/ndarray-layout/tree/Shenghu)
![GitHub repo size](https://img.shields.io/github/repo-size/InfiniTensor/ndarray-layout)
![GitHub code size in bytes](https://img.shields.io/github/languages/code-size/InfiniTensor/ndarray-layout)

Expand All @@ -12,4 +13,43 @@
![GitHub contributors](https://img.shields.io/github/contributors/InfiniTensor/ndarray-layout)
![GitHub commit activity](https://img.shields.io/github/commit-activity/m/InfiniTensor/ndarray-layout)

This crate provides definitions and transformations for multi-dimensional array data layouts.
*ndarray-layout* 是一个用于处理多维数组布局的 *Rust* 库,它提供了 `ArrayLayout` 结构体,用于高效管理和操作多维数组的元信息,如形状、步长和偏移量等。这个库在处理多维数组时,提供了灵活且高效的布局管理方式,能够满足不同场景下对数组布局的操作需求。

## 主要功能特点

### 多维数组布局管理

- `ArrayLayout` 结构体支持指定任意维度的数组布局,通过 `new` 方法可以创建具有指定形状、步长和偏移量的布局;
- 提供 `new_contiguous` 方法,用于创建连续的数组布局,支持大端序(`BigEndian`)和小端序(`LittleEndian`)两种存储顺序;

### 元信息访问

- 提供便捷的方法来访问数组布局的元信息,如 `ndim`、`offset`、`shape` 和 `strides` 等;
- 支持计算数组元素的偏移量和数据范围,方便进行内存访问和数据处理;

### 布局操作功能

- 提供多种布局变换方法,如 `index`、`tile`、`transpose`、`merge` 和 `slice` 等,方便对数组布局进行各种变换操作;

## 使用示例

```rust
use ndarray_layout::{ArrayLayout, BroadcastArg};

// 创建一个新的 `ArrayLayout` 实例。
// 形状为 [1, 2, 3],步长为 [12, 4, 1],偏移量为 0。
let layout = ArrayLayout::<3>::new(&[1, 2, 3], &[12, 4, 1], 0);

// 验证初始的形状和步长。
assert_eq!(layout.shape(), &[1, 2, 3]);
assert_eq!(layout.strides(), &[12, 4, 1]);
assert_eq!(layout.offset(), 0);

// 对第 0 维进行广播变换,广播次数为 4。
let broadcasted_layout = layout.broadcast(0, 4);

// 验证广播变换后的形状和步长。
assert_eq!(broadcasted_layout.shape(), &[4, 2, 3]);
assert_eq!(broadcasted_layout.strides(), &[0, 4, 1]);
assert_eq!(broadcasted_layout.offset(), 0);
```
3 changes: 3 additions & 0 deletions src/fmt.rs
Original file line number Diff line number Diff line change
Expand Up @@ -112,4 +112,7 @@ fn test() {

let tensor = Tensor(tensor.0.tile_be(0, &[2, 3]).tile_be(2, &[5, 2]));
println!("{}", tensor);

let tensor = Tensor(ArrayLayout::<4>::with_ndim(0));
println!("{}", tensor);
}
93 changes: 93 additions & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ impl<const N: usize> ArrayLayout<N> {
content.set_stride(i, mul);
mul *= shape[i] as isize;
};
// 大端小端区别在于是否反转
match endian {
Endian::BigEndian => (0..shape.len()).rev().for_each(push),
Endian::LittleEndian => (0..shape.len()).for_each(push),
Expand Down Expand Up @@ -181,6 +182,13 @@ impl<const N: usize> ArrayLayout<N> {
}

/// Calculates the range of data in bytes to determine the location of the memory area that the array needs to access.
///
/// ```rust
/// # use ndarray_layout::ArrayLayout;
/// let layout = ArrayLayout::<4>::new(&[2, 3, 4],&[12, -4, 1], 20);
/// let range = layout.data_range();
/// assert_eq!(range, 12..=35);
/// ```
pub fn data_range(&self) -> RangeInclusive<isize> {
let content = self.content();
let mut start = content.offset();
Expand Down Expand Up @@ -214,6 +222,7 @@ impl<const N: usize> ArrayLayout<N> {
#[inline]
fn ptr_allocated(&self) -> Option<NonNull<usize>> {
const { assert!(N > 0) }
// ndim > N 则 content 是 ptr,否则是元组。
if self.ndim > N {
Some(unsafe { self.content.ptr })
} else {
Expand Down Expand Up @@ -327,3 +336,87 @@ impl Content<true> {
fn layout(ndim: usize) -> Layout {
Layout::array::<usize>(1 + ndim * 2).unwrap()
}

#[test]
fn test_new() {
let layout = ArrayLayout::<4>::new(&[2, 3, 4], &[12, -4, 1], 20);
assert_eq!(layout.offset(), 20);
assert_eq!(layout.shape(), &[2, 3, 4]);
assert_eq!(layout.strides(), &[12, -4, 1]);
assert_eq!(layout.ndim(), 3);
}

#[test]
fn test_new_contiguous_little_endian() {
let layout = ArrayLayout::<4>::new_contiguous(&[2, 3, 4], Endian::LittleEndian, 4);
assert_eq!(layout.offset(), 0);
assert_eq!(layout.shape(), &[2, 3, 4]);
assert_eq!(layout.strides(), &[4, 8, 24]);
}

#[test]
fn test_new_contiguous_big_endian() {
let layout = ArrayLayout::<4>::new_contiguous(&[2, 3, 4], Endian::LittleEndian, 4);
assert_eq!(layout.offset(), 0);
assert_eq!(layout.shape(), &[2, 3, 4]);
assert_eq!(layout.strides(), &[4, 8, 24]);
}

#[test]
#[should_panic(expected = "shape and strides must have the same length")]
fn test_new_invalid_shape_strides_length() {
ArrayLayout::<4>::new(&[2, 3], &[12, -4, 1], 20);
}

#[test]
fn test_to_inline_size() {
let layout = ArrayLayout::<4>::new_contiguous(&[3, 4], Endian::BigEndian, 0);
assert_eq!(size_of_val(&layout), (2 * 4 + 2) * size_of::<usize>());
let layout = layout.to_inline_size::<2>();
assert_eq!(size_of_val(&layout), (2 * 2 + 2) * size_of::<usize>());
}

#[test]
fn test_num_elements() {
let layout = ArrayLayout::<4>::new_contiguous(&[2, 3, 4], Endian::BigEndian, 20);
assert_eq!(layout.num_elements(), 24);
}

#[test]
fn test_element_offset_little_endian() {
let layout = ArrayLayout::<4>::new_contiguous(&[2, 3, 4], Endian::LittleEndian, 4);
assert_eq!(layout.element_offset(22, Endian::LittleEndian), 88);
}

#[test]
fn test_element_offset_big_endian() {
let layout = ArrayLayout::<4>::new_contiguous(&[2, 3, 4], Endian::BigEndian, 4);
assert_eq!(layout.element_offset(22, Endian::BigEndian), 88);
}

#[test]
fn test_data_range_positive_strides() {
let layout = ArrayLayout::<4>::new_contiguous(&[2, 3, 4], Endian::LittleEndian, 4);
let range = layout.data_range();
assert_eq!(range, 0..=92); // 0 + 2*4 + 3*8 + 4*24 = 92
}

#[test]
fn test_data_range_mixed_strides() {
let layout = ArrayLayout::<4>::new(&[2, 3, 4], &[12, -4, 0], 20);
let range = layout.data_range();
assert_eq!(range, 12..=32);
}

#[test]
fn test_clone_and_eq() {
let layout1 = ArrayLayout::<4>::new(&[2, 3, 4], &[12, -4, 1], 20);
let layout2 = layout1.clone();
assert!(layout1.eq(&layout2));
}

#[test]
fn test_drop() {
let layout = ArrayLayout::<4>::new(&[2, 3, 4], &[12, -4, 1], 20);
drop(layout);
}
10 changes: 9 additions & 1 deletion src/transform/broadcast.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use crate::ArrayLayout;

/// 索引变换参数
/// 广播变换参数
#[derive(Clone, PartialEq, Eq, Debug)]
pub struct BroadcastArg {
/// 广播的轴。
Expand Down Expand Up @@ -35,3 +35,11 @@ impl<const N: usize> ArrayLayout<N> {
ans
}
}

#[test]
fn test_broadcast() {
let layout = ArrayLayout::<3>::new(&[1, 5, 2], &[10, 2, 1], 0).broadcast(0, 10);
assert_eq!(layout.shape(), &[10, 5, 2]);
assert_eq!(layout.strides(), &[0, 2, 1]);
assert_eq!(layout.offset(), 0);
}
15 changes: 15 additions & 0 deletions src/transform/index.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,4 +78,19 @@ fn test() {
assert_eq!(layout.shape(), &[2, 4]);
assert_eq!(layout.strides(), &[12, 1]);
assert_eq!(layout.offset(), 12);

let layout = ArrayLayout::<4>::new(&[2, 3, 4], &[12, -4, 1], 20);
let layout = layout.index_many(&[]);
assert_eq!(layout.shape(), &[2, 3, 4]);
assert_eq!(layout.strides(), &[12, -4, 1]);
assert_eq!(layout.offset(), 20);

let layout = ArrayLayout::<4>::new(&[2, 3, 4], &[12, -4, 1], 20);
let layout = layout.index_many(&[
IndexArg { axis: 0, index: 1 },
IndexArg { axis: 1, index: 2 },
]);
assert_eq!(layout.shape(), &[4]);
assert_eq!(layout.strides(), &[1]);
assert_eq!(layout.offset(), 24);
}
70 changes: 66 additions & 4 deletions src/transform/merge.rs
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,11 @@ impl<const N: usize> ArrayLayout<N> {
let shape = content.shape();
let strides = content.strides();

let merged = args.iter().map(|arg| arg.len).sum::<usize>();
let merged = args.iter().map(|arg| arg.len.max(1)).sum::<usize>();
if merged == args.len() {
return Some(self.clone());
}

let mut ans = Self::with_ndim(self.ndim + args.len() - merged);

let mut content = ans.content_mut();
Expand All @@ -93,7 +97,7 @@ impl<const N: usize> ArrayLayout<N> {
let &MergeArg { start, len, endian } = arg;
let end = start + len;

if len == 0 {
if len < 2 {
continue;
}

Expand All @@ -109,6 +113,9 @@ impl<const N: usize> ArrayLayout<N> {
_ => pairs.push((d, s)),
}
}

last_end = end;

if pairs.is_empty() {
push(1, 0);
continue;
Expand All @@ -131,7 +138,6 @@ impl<const N: usize> ArrayLayout<N> {
}

push(d, *s);
last_end = end;
}
for j in last_end..shape.len() {
push(shape[j], strides[j]);
Expand All @@ -142,11 +148,67 @@ impl<const N: usize> ArrayLayout<N> {
}

#[test]
fn test_merge() {
fn test_merge_return_none() {
let layout = ArrayLayout::<3>::new(&[16, 4, 2], &[8, 4, 1], 0).merge_be(0, 3);
assert!(layout.is_none());
}

#[test]
fn test_merge_pairs_empyt() {
let layout = ArrayLayout::<3>::new(&[1, 1, 1], &[1, 1, 1], 0)
.merge_be(0, 2)
.unwrap();
assert_eq!(layout.shape(), &[1, 1]);
assert_eq!(layout.strides(), &[0, 1]);
assert_eq!(layout.offset(), 0);
}

#[test]
fn test_merge_be_example() {
let layout = ArrayLayout::<3>::new(&[16, 1, 4], &[16, 768, 4], 0)
.merge_be(0, 2)
.unwrap();
assert_eq!(layout.shape(), &[16, 4]);
assert_eq!(layout.strides(), &[16, 4]);
assert_eq!(layout.offset(), 0);
}

#[test]
fn test_merge_le_example() {
let layout = ArrayLayout::<3>::new(&[4, 3, 2], &[1, 4, 12], 0);
let merged_layout = layout.merge_le(0, 3).unwrap();

assert_eq!(merged_layout.shape(), &[24]);
assert_eq!(merged_layout.strides(), &[1]);
assert_eq!(merged_layout.offset(), 0);
}

#[test]
fn test_merge_len_zero() {
let layout = ArrayLayout::<3>::new(&[4, 3, 2], &[1, 4, 12], 0);
let merged_layout = layout.merge_le(0, 0).unwrap();

assert_eq!(merged_layout.shape(), &[4, 3, 2]);
assert_eq!(merged_layout.strides(), &[1, 4, 12]);
assert_eq!(merged_layout.offset(), 0);
}

#[test]
fn test_partial_merge() {
let layout = ArrayLayout::<4>::new(&[2, 3, 4, 5], &[60, 20, 5, 1], 0);
let merged_layout = layout.merge_be(1, 2).unwrap();

assert_eq!(merged_layout.shape(), &[2, 12, 5]);
assert_eq!(merged_layout.strides(), &[60, 5, 1]);
assert_eq!(merged_layout.offset(), 0);
}

#[test]
fn test_merge_free_example() {
let layout = ArrayLayout::<3>::new(&[3, 2, 4], &[4, 12, 1], 0);
let merged_layout = layout.merge_free(0, 3).unwrap();

assert_eq!(merged_layout.shape(), &[24]);
assert_eq!(merged_layout.strides(), &[1]);
assert_eq!(merged_layout.offset(), 0);
}
Loading