Revamping Documentation: Enhancing, Restructuring and Covering More Topics (#581)

* refactor docs.

* add manual and automatic derivation.

* integration with zio streams.

* add more packages to installation section.

* add comment for fail constructor.

* primitives.

* transforming schemas.

* sequence section.

* map collection.

* schema for set collection.

* record section.

* add case class section.

* refactor.

* generic record.

* either.

* enumeration.

* chunk and vector.

* fail.

* tuples.

* update readme.

* refactor sidebars.

* add more resources for zio schema.

* getting the default value of a schema.

* sidebar label for operations.

* reorder sidebar items.

* generate ordering for schemas.

* fix mdoc errors.

* diffing and patching.

* automatic migration.

* schema serialization.

* fix typo.

* add more resources.

* mapping dto to domain object.

* example for Schema#migrate method.

* improve sentences.

* add nuttycombe talk to resource section.

* dynamic data representation.

* dynamic value migration.

* schema migration.

* derive ordering.

* separate article for getting the default value.

* separate article for diffing and patching.

* separate other articles.

* validation section.

* update dynamic data representation article.

* reified optics.

* avro codecs.

* add bson codecs.

* improve apache avro doc.

* add zio-schema-bson to doc's dependencies.

* add json codec article.

* add message pack section.

* add protobuf section.

* update message pack.

* add apache thrift section to codecs.

* refactor.

* refactor.

* organize sidebar.

* overview of all operations.

* update introduction section.

* update readme.

* update readme.

* update readme.

Author: khajavi

README.md

@@ -10,76 +10,93 @@
 
 ## Introduction
 
-Schema is a structure of a data type. ZIO Schema reifies the concept of structure for data types. It makes a high-level description of any data type and makes them as first-class values.
+ZIO Schema helps us to solve some of the most common problems in distributed computing, such as serialization, deserialization, and data migration.
+
+It turns a compiled-time construct (the type of a data structure) into a runtime construct (a value that can be read, manipulated, and composed at runtime). A schema is a structure of a data type. ZIO Schema reifies the concept of structure for data types. It makes a high-level description of any data type and makes them first-class values.
 
 Creating a schema for a data type helps us to write codecs for that data type. So this library can be a host of functionalities useful for writing codecs and protocols like JSON, Protobuf, CSV, and so forth.
 
-With schema descriptions that can be automatically derived for case classes and sealed traits, _ZIO Schema_ will be going to provide powerful features for free (Note that the project is in the development stage and all these features are not supported yet):
+## What Problems Does ZIO Schema Solve?
+
+With schema descriptions that can be automatically derived for case classes and sealed traits, _ZIO Schema_ will be going to provide powerful features for free:
 
-- Codecs for any supported protocol (JSON, protobuf, etc.), so data structures can be serialized and deserialized in a principled way
-- Diffing, patching, merging, and other generic-data-based operations
-- Migration of data structures from one schema to another compatible schema
-- Derivation of arbitrary type classes (`Eq`, `Show`, `Ord`, etc.) from the structure of the data
+1. Metaprogramming without macros, reflection, or complicated implicit derivations.
+    1. Creating serialization and deserialization codecs for any supported protocol (JSON, Protobuf, etc.)
+    2. Deriving standard type classes (`Eq`, `Show`, `Ordering`, etc.) from the structure of the data
+    3. Default values for data types
+2. Automate ETL (Extract, Transform, Load) pipelines
+    1. Diffing: diffing between two values of the same type
+    2. Patching: applying a diff to a value to update it
+    3. Migration: migrating values from one type to another
+3. Computations as data: Not only we can turn types into values, but we can also turn computations into values. This opens up a whole new world of possibilities concerning distributed computing.
 
 When our data structures need to be serialized, deserialized, persisted, or transported across the wire, then _ZIO Schema_ lets us focus on data modeling and automatically tackle all the low-level, messy details for us.
 
-_ZIO Schema_ is used by a growing number of ZIO libraries, including _ZIO Flow_, _ZIO Redis_, _ZIO Web_, _ZIO SQL_ and _ZIO DynamoDB_.
+_ZIO Schema_ is used by a growing number of ZIO libraries, including [ZIO Flow](https://zio.dev/zio-flow), [ZIO Redis](https://zio-redis), [ZIO SQL](https://zio.dev/zio-sql) and [ZIO DynamoDB](https://zio.dev/zio-dynamodb).
 
 ## Installation
 
 In order to use this library, we need to add the following lines in our `build.sbt` file:
 
 ```scala
-libraryDependencies += "dev.zio" %% "zio-schema"          % "0.4.13"
-libraryDependencies += "dev.zio" %% "zio-schema-bson"     % "0.4.13"
-libraryDependencies += "dev.zio" %% "zio-schema-json"     % "0.4.13"
-libraryDependencies += "dev.zio" %% "zio-schema-protobuf" % "0.4.13"
-
-// Required for automatic generic derivation of schemas
-libraryDependencies += "dev.zio" %% "zio-schema-derivation" % "0.4.13",
+libraryDependencies += "dev.zio" %% "zio-schema"          % "0.4.14"
+libraryDependencies += "dev.zio" %% "zio-schema-avro"     % "0.4.14"
+libraryDependencies += "dev.zio" %% "zio-schema-bson"     % "0.4.14"
+libraryDependencies += "dev.zio" %% "zio-schema-json"     % "0.4.14"
+libraryDependencies += "dev.zio" %% "zio-schema-msg-pack" % "0.4.14"
+libraryDependencies += "dev.zio" %% "zio-schema-protobuf" % "0.4.14"
+libraryDependencies += "dev.zio" %% "zio-schema-thrift"   % "0.4.14"
+libraryDependencies += "dev.zio" %% "zio-schema-zio-test" % "0.4.14"
+
+// Required for the automatic generic derivation of schemas
+libraryDependencies += "dev.zio" %% "zio-schema-derivation" % "0.4.14"
 libraryDependencies += "org.scala-lang" % "scala-reflect"  % scalaVersion.value % "provided"
 ```
 
 ## Example
 
-In this simple example first, we create a schema for `Person` and then run the _diff_ operation on two instances of the `Person` data type, and finally we encode a Person instance using _Protobuf_ protocol:
+In this simple example first, we create a schema for `Person` and then run the _diff_ operation on two instances of the `Person` data type, and finally, we encode a Person instance using _Protobuf_ protocol:
 
 ```scala
-import zio.console.putStrLn
-import zio.schema.codec.ProtobufCodec._
+import zio._
+import zio.stream._
+import zio.schema.codec.{BinaryCodec, ProtobufCodec}
 import zio.schema.{DeriveSchema, Schema}
-import zio.stream.ZStream
-import zio.{Chunk, ExitCode, URIO}
 
-final case class Person(name: String, age: Int, id: String)
+final case class Person(name: String, age: Int)
+
 object Person {
-  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
+  implicit val schema: Schema[Person]    = DeriveSchema.gen
+  val protobufCodec: BinaryCodec[Person] = ProtobufCodec.protobufCodec
 }
 
-Person.schema
-
-import zio.schema.syntax._
-
-Person("Alex", 31, "0123").diff(Person("Alex", 31, "124"))
+object Main extends ZIOAppDefault {
+  def run =
+    ZStream
+      .succeed(Person("John", 43))
+      .via(Person.protobufCodec.streamEncoder)
+      .runCollect
+      .flatMap(x =>
+        Console.printLine(s"Encoded data with protobuf codec: ${toHex(x)}")
+      )
+
+  def toHex(chunk: Chunk[Byte]): String =
+    chunk.map("%02X".format(_)).mkString
+}
+```
 
-def toHex(chunk: Chunk[Byte]): String =
-  chunk.toArray.map("%02X".format(_)).mkString
+Here is the output of running the above program:
 
-zio.Runtime.default.unsafe.run(
-  ZStream
-    .succeed(Person("Thomas", 23, "2354"))
-    .transduce(
-      encoder(Person.schema)
-    )
-    .runCollect
-    .flatMap(x => putStrLn(s"Encoded data with protobuf codec: ${toHex(x)}"))
-).getOrThrowFiberFailure()
+```scala
+Encoded data with protobuf codec: 0A044A6F686E102B
 ```
 
-
 ## Resources
 
-- [Zymposium - ZIO Schema](https://www.youtube.com/watch?v=GfNiDaL5aIM) by John A. De Goes, Adam Fraser and Kit Langton (May 2021)
+- [Zymposium - ZIO Schema](https://www.youtube.com/watch?v=GfNiDaL5aIM) by John A. De Goes, Adam Fraser, and Kit Langton (May 2021)
+- [ZIO SCHEMA: A Toolkit For Functional Distributed Computing](https://www.youtube.com/watch?v=lJziseYKvHo&t=481s) by Dan Harris (Functional Scala 2021)
+- [Creating Declarative Query Plans With ZIO Schema](https://www.youtube.com/watch?v=ClePN4P9_pg) by Dan Harris (ZIO World 2022)
+- [Describing Data...with free applicative functors (and more)](https://www.youtube.com/watch?v=oRLkb6mqvVM) by Kris Nuttycombe (Scala World) on the idea behind the [xenomorph](https://github.com/nuttycom/xenomorph) library
 
 ## Documentation
 

build.sbt

@@ -341,4 +341,14 @@ lazy val docs = project
          |sbt test
          |```""".stripMargin
   )
+  .dependsOn(
+    zioSchemaJVM,
+    zioSchemaProtobufJVM,
+    zioSchemaJsonJVM,
+    zioSchemaOpticsJVM,
+    zioSchemaAvroJVM,
+    zioSchemaBsonJVM,
+    zioSchemaMsgPackJVM,
+    zioSchemaThriftJVM
+  )
   .enablePlugins(WebsitePlugin)

docs/automatic-schema-derivation.md

@@ -0,0 +1,91 @@
+---
+id: automatic-schema-derivation
+title: "Automatic Schema Derivation"
+---
+
+Automatic schema derivation is the process of generating schema definitions for data types automatically, without the need to manually write them. It allows us to generate the schema for a data type based on its structure and annotations.
+
+Instead of manually specifying the schema for each data type, we can rely on automatic schema derivation to generate the schema for us. This approach can save time and reduce the potential for errors, especially when dealing with complex data models.
+
+By leveraging reflection and type introspection using macros, automatic schema derivation analyzes the structure of the data type and its fields, including their names, types, and annotations. It then generates the corresponding schema definition based on this analysis.
+
+ZIO streamlines schema derivation through its `zio-schema-derivation` package, which utilizes the capabilities of Scala macros to automatically derive schemas. In order to use automatic schema derivation, we neeed to add the following line to our `build.sbt` file:
+
+```scala
+libraryDependencies += "dev.zio" %% "zio-schema-derivation" % @VERSION@
+```
+
+Once again, let's revisit our domain models:
+
+```scala mdoc:compile-only
+final case class Person(name: String, age: Int)
+
+sealed trait PaymentMethod
+
+object PaymentMethod {
+  final case class CreditCard(number: String, expirationMonth: Int, expirationYear: Int) extends PaymentMethod
+  final case class WireTransfer(accountNumber: String, bankCode: String) extends PaymentMethod
+}
+
+final case class Customer(person: Person, paymentMethod: PaymentMethod)
+```
+
+We can easily use auto derivation to create schemas:
+
+```scala
+import zio.schema._
+import zio.schema.codec._
+
+final case class Person(name: String, age: Int)
+
+object Person {
+  implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
+}
+
+sealed trait PaymentMethod
+
+object PaymentMethod {
+
+  implicit val schema: Schema[PaymentMethod] =
+    DeriveSchema.gen[PaymentMethod]
+
+  final case class CreditCard(
+      number: String,
+      expirationMonth: Int,
+      expirationYear: Int
+  ) extends PaymentMethod
+
+  final case class WireTransfer(accountNumber: String, bankCode: String)
+      extends PaymentMethod
+}
+
+final case class Customer(person: Person, paymentMethod: PaymentMethod)
+
+object Customer {
+  implicit val schema: Schema[Customer] = DeriveSchema.gen[Customer]
+}
+```
+
+Now we can write an example that demonstrates a roundtrip test for protobuf codecs:
+
+```scala
+// Create a customer instance
+val customer =
+  Customer(
+    person = Person("John Doe", 42),
+    paymentMethod = PaymentMethod.CreditCard("1000100010001000", 6, 2024)
+  )
+
+// Create binary codec from customer 
+val customerCodec: BinaryCodec[Customer] =
+  ProtobufCodec.protobufCodec[Customer]
+
+// Encode the customer object
+val encodedCustomer: Chunk[Byte] = customerCodec.encode(customer)
+
+// Decode the byte array back to the person instance
+val decodedCustomer: Either[DecodeError, Customer] =
+  customerCodec.decode(encodedCustomer)
+
+assert(Right(customer) == decodedCustomer)
+```