Fixed bugs in adding deps to pyproject.toml, packaging deps, and adde…

…d dev deps

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## v0.1.3
+- Added support for dev dependencies
+- Fixed a bug where dependencies weren't being set up with the `package` command
+
 ## v0.1.2
 - Added support for installing Python on most Linux distros
 - Wheel is now installed directly, instead of with Pip; should only be dependent on

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "pyflow"
-version = "0.1.2"
+version = "0.1.3"
 authors = ["David O'Connor <david.alan.oconnor@gmail.com>"]
 description = "A modern Python dependency manager"
 license = "MIT"

README.md

@@ -8,6 +8,8 @@
 
 This tool manages Python installations and dependencies.
 
+![Poetry Install](https://raw.githubusercontent.com/david-oconnor/pyflow/master/assets/install.gif)
+
 **Goals**: Make using and publishing Python projects as simple as possible. Understanding
 Python environments shouldn't be required to use dependencies safely. We're attempting
 to fix each stumbling block in the Python workflow, so that it's as elegant
@@ -24,11 +26,11 @@ and [Pep 518 (pyproject.toml)](https://www.python.org/dev/peps/pep-0518/), and s
 
 ## Installation
 - **Windows, Ubuntu, or Debian:** Download and run
-[this installer](https://github.com/David-OConnor/pyflow/releases/download/0.1.2/pyflow-0.1.2-x86_64.msi)
+[this installer](https://github.com/David-OConnor/pyflow/releases/download/0.1.3/pyflow-0.1.3-x86_64.msi)
 or
-[this deb](https://github.com/David-OConnor/pyflow/releases/download/0.1.2/pyflow_0.1.2_amd64.deb) .
+[this deb](https://github.com/David-OConnor/pyflow/releases/download/0.1.3/pyflow_0.1.3_amd64.deb) .
 
-- **A different Linux distro:** Download this [standalone binary](https://github.com/David-OConnor/pyflow/releases/download/0.1.2/pyflow)
+- **A different Linux distro:** Download this [standalone binary](https://github.com/David-OConnor/pyflow/releases/download/0.1.3/pyflow)
  and place it somewhere
 accessible by the system PATH. For example, `/usr/bin`.
 
@@ -57,32 +59,46 @@ to run one-off Python files that aren't attached to a project, but have dependen
 
 
 ## Why add another Python manager?
-`Pipenv` and `Poetry` both address part of Pyflow's *raison d'être*.
- Some reasons why this is different:
+`Pipenv`, `Poetry`, and `Pyenv` address parts of 
+Pyflow's *raison d'être*, but expose stumbling blocks that may frustrate new users, 
+both when installing and using.  Some reasons why this is different:
 
- - It automatically manages Python installations and environments. You specify a Python version
+- It automatically manages Python installations and environments. You specify a Python version
  in `pyproject.toml` (if ommitted, it asks), and ensures that version is used. 
  If the version's not installed, Pyflow downloads a binary, and uses that.
  If multiple installations are found for that version, it asks which to use.
+ `Pyenv` can be used to install Python, but only if your system is configured in a certain way: 
+ I don’t think expecting a user’s computer to compile Python is reasonable.
 
 - By not using Python to install or run, it remains environment-agnostic. 
 This is important for making setup and use as simple and decison-free as
  possible. It's especially important on Linux, where there may be several versions
 of Python installed, with different versions and access levels. This avoids
 complications, especially for new users. It's common for Python-based CLI tools
 to not run properly when installed from `pip` due to the `PATH` or user directories
-not being configured in the expected way.
+not being configured in the expected way. Pipenv’s installation 
+instructions are confusing, and may result in it not working correctly.
 
 - Its dependency resolution and locking is faster due to using a cached
 database of dependencies, vice downloading and checking each package, or relying
 on the incomplete data available on the [pypi warehouse](https://github.com/pypa/warehouse).
+Pipenv’s resolution in particular may be prohibitively-slow on weak internet connections.
 
 - It keeps dependencies in the project directory, in `__pypackages__`. This is subtle, 
 but reinforces the idea that there's
 no hidden state.
 
-- It will always use the specified version of Python. This is a notable limitation in `Poetry`; it
+- It will always use the specified version of Python. This is a notable limitation in `Poetry`; Poetry
 may pick the wrong installation (eg Python2 vice Python3), with no obvious way to change it.
+Poetry allows projects to specify version, but neither selects, 
+nor provides a way to select the right one. If it chooses the wrong one, it will 
+install the wrong environment, and produce a confusing 
+error message. This can be worked around using `Pyenv`, but neither the poetry docs 
+nor error message provide guidance 
+on this. This adds friction to the workflow and may confuse new users, as it occurs 
+by default on popular linux distros like Ubuntu. Additionally, `pyenv's` docs are 
+confusing: It's not obvious how to install it, what operating systems
+it's compatible with, or what additional dependencies are required.
 
 - Multiple versions of a dependency can be installed, allowing resolution
 of conflicting sub-dependencies. (ie: Your package requires `Dep A>=1.0` and `Dep B`.
@@ -166,7 +182,9 @@ diffeqpy = "1.1.0"
 The `[tool.pyflow]` section is used for metadata. The only required item in it is
  `py_version`, unless
 building and distributing a package. The `[tool.pyflow.dependencies]` section
-contains all dependencies, and is an analog to `requirements.txt`. 
+contains all dependencies, and is an analog to `requirements.txt`. You can specify
+developer dependencies in the `[tool.pyflow.dev-dependencies]` section. These
+won't be packed or published, but will be installed locally.
 
 You can specify `extra` dependencies, which will only be installed when passing
 explicit flags to `pyflow install`, or when included in another project with the appropriate
@@ -304,7 +322,6 @@ check for resolutions, then vary children as-required down the hierarchy. We don
 - Installing from sources other than `pypi` (eg repos, paths)
 - The lock file is missing some info like hashes
 - Adding a dependency via the CLI with a specific version constraint, or extras.
-- Developer dependencies
 - Packaging and publishing projects that use compiled extensions
 - Dealing with multiple-installed-versions of a dependency that uses importlib
 or dynamic imports
@@ -339,6 +356,10 @@ package_url = "https://upload.pypi.org/legacy/"
 numpy = "^1.16.4"
 manim = "0.1.8"
 ipython = {version = "^7.7.0", extras=["qtconsole"]}
+
+
+[tool.pyflow.dev-dependencies]
+black = "^18.0"
 ```
 `package_url` is used to determine which package repository to upload to. If ommitted, 
 `Pypi test` is used (`https://test.pypi.org/legacy/`).

src/build.rs

@@ -61,6 +61,8 @@ fn create_dummy_setup(cfg: &crate::Config, filename: &str) {
         keywords.push_str(kw);
     }
 
+    let deps: Vec<String> = cfg.reqs.iter().map(|r| r.to_setup_py_string()).collect();
+
     let data = format!(
         r#"import setuptools
  
@@ -81,6 +83,7 @@ setuptools.setup(
     keywords="{}",
     classifiers={},
     python_requires="{}",
+    install_requires={},
 )
 "#,
         //            entry_points={{
@@ -98,6 +101,7 @@ setuptools.setup(
         serialize_py_list(&cfg.classifiers),
         //        serialize_py_list(&cfg.console_scripts),
         cfg.python_requires.unwrap_or_else(|| "".into()),
+        serialize_py_list(&deps),
         // todo:
         //            extras_require="{}",
         //        match cfg.extras {

src/dep_types.rs

@@ -858,10 +858,21 @@ impl Req {
             ),
         }
     }
-    //    /// Return true if other is a subset of self.
-    //    fn _fully_contains(&self, other: &Self) -> bool {
-    //
-    //    }
+
+    /// Format for setup.py
+    pub fn to_setup_py_string(&self) -> String {
+        format!(
+            "{}{}",
+            self.name,
+            self.constraints
+                .iter()
+                .map(|c| c.to_string(false, true))
+                .collect::<Vec<String>>()
+                .join(",")
+        )
+        .replace("^", ">")
+        .replace("~", ">") // todo: Sloppy, but perhaps the best way.
+    }
 }
 
 impl fmt::Display for Req {

src/files.rs

@@ -125,43 +125,43 @@ pub fn add_reqs_to_cfg(filename: &str, added: &[Req]) {
 
     // We collect lines here so we can start the index at a non-0 point.
     let lines_vec: Vec<&str> = data.lines().collect();
+    let mut insertion_pt = 0;
 
     for (i, line) in data.lines().enumerate() {
-        result.push_str(line);
-        result.push_str("\n");
-        if line == "[tool.pyflow.dependencies]" {
+        if &line.replace(" ", "") == "[tool.pyflow.dependencies]" {
             in_dep = true;
+            continue;
+        }
 
-            if i != lines_vec.len() - 1 {
-                // If the last line's the start of dependencies section, don't move on;
-                // we'll add now.
-                continue;
-            }
+        // We've found the end of the dependencies section.
+        if in_dep && (sect_re.is_match(line) || i == lines_vec.len() - 1) {
+            insertion_pt = i - 2;
+            break;
         }
+    }
 
-        if in_dep {
-            let mut ready_to_insert = true;
-            // Check if this is the last non-blank line in the dependencies section.
-            for i2 in i..lines_vec.len() {
-                let line2 = lines_vec[i2];
-                // We've hit the end of the section or file without encountering a non-empty line.
-                if sect_re.is_match(line2) || i2 == lines_vec.len() - 1 {
-                    break;
-                }
-                if !line2.is_empty() {
-                    // We haven't hit the end of the section yet; don't add the new reqs here.
-                    ready_to_insert = false;
-                    break;
-                }
-            }
-            if ready_to_insert {
-                for req in added {
-                    result.push_str(&req.to_cfg_string());
-                    result.push_str("\n");
-                }
+    for (i, line) in data.lines().enumerate() {
+        result.push_str(line);
+        result.push_str("\n");
+
+        if i == insertion_pt {
+            for req in added {
+                result.push_str(&req.to_cfg_string());
+                result.push_str("\n");
             }
         }
     }
+    if !in_dep {
+        // todo: A bit of an awkward way to handle.
+        result.push_str("[tool.pyflow.dependencies]");
+        result.push_str("\n");
+        for req in added {
+            result.push_str(&req.to_cfg_string());
+            result.push_str("\n");
+        }
+    }
+
+    //        }
 
     fs::write(filename, result)
         .expect("Unable to write pyproject.toml while attempting to add a dependency");

src/main.rs

@@ -398,7 +398,7 @@ impl Config {
                 result.reqs = Self::parse_deps(deps);
             }
             if let Some(deps) = pf.dev_dependencies {
-                result.reqs = Self::parse_deps(deps);
+                result.dev_reqs = Self::parse_deps(deps);
             }
         }
 
@@ -500,6 +500,8 @@ package_url = "https://test.pypi.org/legacy/"
 
 
 [tool.pyflow.dependencies]
+
+[tool.pyflow.dev-dependencies]
 "##,
         name
     );
@@ -1374,12 +1376,23 @@ fn main() {
             files::remove_reqs_from_cfg(cfg_filename, &removed_reqs);
 
             // Filter reqs here instead of re-reading the config from file.
-            let updated_reqs: Vec<Req> = cfg
+            let mut updated_reqs: Vec<Req> = cfg
                 .reqs
                 .into_iter()
                 .filter(|req| !removed_reqs.contains(&req.name))
                 .collect();
 
+            // todo: DRY
+            let updated_dev_reqs: Vec<Req> = cfg
+                .dev_reqs
+                .into_iter()
+                .filter(|req| !removed_reqs.contains(&req.name))
+                .collect();
+
+            for dev_req in updated_dev_reqs.into_iter() {
+                updated_reqs.push(dev_req);
+            }
+
             sync(
                 &bin_path,
                 &lib_path,

src/py_versions.rs

@@ -168,10 +168,10 @@ fn download(py_install_path: &Path, version: &Version) {
             "Linux distro",
             &[
                 (
-                    "2018 or newer (Ubuntu ≥ 18.04, Debian, Kali etc)".to_owned(),
+                    "2018 or newer (Ubuntu≥18.04, Debian≥10, Suse≥15, Kali, etc)".to_owned(),
                     Os::Ubuntu,
                 ),
-                ("Older (Centos, Redhat, Suse etc)".to_owned(), Os::Centos),
+                ("Older (Centos, Redhat)".to_owned(), Os::Centos),
             ],
             false,
         );
@@ -381,7 +381,7 @@ pub fn create_venv(
             _ => {
                 //                let r = prompt_alias(&aliases);
                 let r = util::prompt_list(
-                    "Found multiple compatible Python aliases. Please enter the number associated with the one you'd like to use for this project:",
+                    "Found multiple compatible Python versions. Please enter the number associated with the one you'd like to use:",
                     "Python alias",
                     &aliases,
                     true,
@@ -428,7 +428,12 @@ pub fn create_venv(
         fs::create_dir_all(&lib_path).expect("Problem creating __pypackages__ directory");
     }
 
-    println!("Setting up Python environment...");
+    #[cfg(target_os = "windows")]
+    println!("🐍🐍🐍🐍Setting up Python environment...");
+    #[cfg(target_os = "linux")]
+    println!("🐍Setting up Python environment..."); // Beware! Snake may be invisible.
+    #[cfg(target_os = "macos")]
+    println!("🐍Setting up Python environment...");
 
     // For an alias on the PATH
     if let Some(alias) = alias {

src/util.rs

@@ -234,7 +234,12 @@ pub fn merge_reqs(added: &[String], cfg: &crate::Config, cfg_filename: &str) ->
             // return true if the added req's not in the cfg reqs, or if it is
             // and the version's different.
             let mut add = true;
-            for cr in cfg.reqs.iter() {
+            let mut reqs = cfg.reqs.clone();
+            for dev_req in cfg.dev_reqs.clone().into_iter() {
+                reqs.push(dev_req);
+            }
+
+            for cr in reqs.iter() {
                 if cr == ar
                     || (cr.name.to_lowercase() == ar.name.to_lowercase()
                         && ar.constraints.is_empty())