Skip to content

Commit

Permalink
sinusoidal positional encoding implemented as util for embedded layer…
Browse files Browse the repository at this point in the history
… in the future
  • Loading branch information
JakubSchwenkbeck committed Dec 4, 2024
1 parent f4ce2eb commit a209353
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions src/math/positional_encoding.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
pub fn sinusoidal_pos_encoding(pos: usize, index: usize, embedding: usize) -> f32 {
let divisor = 10000f32.powf(2.0 * (index / embedding) as f32);

if index % 2 == 0 {
(pos as f32 / divisor).sin()
} else {
(pos as f32 / divisor).cos()
}
}

0 comments on commit a209353

Please sign in to comment.