400 lines
12 KiB
Markdown
400 lines
12 KiB
Markdown
# Internet-ML: Allowing ML to connect to the internet
|
||
|
||
<div align="center">
|
||
|
||
[](https://github.com/internet-ml/internet-ml/actions?query=workflow%3Abuild)
|
||
[](https://pypi.org/project/internet-ml/)
|
||
[](https://github.com/internet-ml/internet-ml/pulls?utf8=%E2%9C%93&q=is%3Apr%20author%3Aapp%2Fdependabot)
|
||
|
||
[](https://github.com/psf/black)
|
||
[](https://github.com/PyCQA/bandit)
|
||
[](https://github.com/internet-ml/internet-ml/blob/master/.pre-commit-config.yaml)
|
||
[](https://github.com/internet-ml/internet-ml/releases)
|
||
[](https://github.com/internet-ml/internet-ml/blob/master/LICENSE)
|
||
|
||
|
||
</div>
|
||
|
||
## Very first steps
|
||
|
||
### Initialize your code
|
||
|
||
1. Initialize `git` inside your repo:
|
||
|
||
```bash
|
||
cd internet-ml && git init
|
||
```
|
||
|
||
2. If you don't have `Poetry` installed run:
|
||
|
||
```bash
|
||
make poetry-download
|
||
```
|
||
|
||
3. Initialize poetry and install `pre-commit` hooks:
|
||
|
||
```bash
|
||
make install
|
||
make pre-commit-install
|
||
```
|
||
|
||
4. Run the codestyle:
|
||
|
||
```bash
|
||
make codestyle
|
||
```
|
||
|
||
5. Upload initial code to GitHub:
|
||
|
||
```bash
|
||
git add .
|
||
git commit -m ":tada: Initial commit"
|
||
git branch -M main
|
||
git remote add origin https://github.com/internet-ml/internet-ml.git
|
||
git push -u origin main
|
||
```
|
||
|
||
### Set up bots
|
||
|
||
- Set up [Dependabot](https://docs.github.com/en/github/administering-a-repository/enabling-and-disabling-version-updates#enabling-github-dependabot-version-updates) to ensure you have the latest dependencies.
|
||
- Set up [Stale bot](https://github.com/apps/stale) for automatic issue closing.
|
||
|
||
### Poetry
|
||
|
||
Want to know more about Poetry? Check [its documentation](https://python-poetry.org/docs/).
|
||
|
||
<details>
|
||
<summary>Details about Poetry</summary>
|
||
<p>
|
||
|
||
Poetry's [commands](https://python-poetry.org/docs/cli/#commands) are very intuitive and easy to learn, like:
|
||
|
||
- `poetry add numpy@latest`
|
||
- `poetry run pytest`
|
||
- `poetry publish --build`
|
||
|
||
etc
|
||
|
||
</p>
|
||
</details>
|
||
|
||
### Building and releasing your package
|
||
|
||
Building a new version of the application contains steps:
|
||
|
||
- Bump the version of your package `poetry version <version>`. You can pass the new version explicitly, or a rule such as `major`, `minor`, or `patch`. For more details, refer to the [Semantic Versions](https://semver.org/) standard.
|
||
- Make a commit to `GitHub`.
|
||
- Create a `GitHub release`.
|
||
- And... publish 🙂 `poetry publish --build`
|
||
|
||
## 🎯 What's next
|
||
|
||
Well, that's up to you 💪🏻. I can only recommend the packages and articles that helped me.
|
||
|
||
- [`Typer`](https://github.com/tiangolo/typer) is great for creating CLI applications.
|
||
- [`Rich`](https://github.com/willmcgugan/rich) makes it easy to add beautiful formatting in the terminal.
|
||
- [`Pydantic`](https://github.com/samuelcolvin/pydantic/) – data validation and settings management using Python type hinting.
|
||
- [`Loguru`](https://github.com/Delgan/loguru) makes logging (stupidly) simple.
|
||
- [`tqdm`](https://github.com/tqdm/tqdm) – fast, extensible progress bar for Python and CLI.
|
||
- [`IceCream`](https://github.com/gruns/icecream) is a little library for sweet and creamy debugging.
|
||
- [`orjson`](https://github.com/ijl/orjson) – ultra fast JSON parsing library.
|
||
- [`Returns`](https://github.com/dry-python/returns) makes you function's output meaningful, typed, and safe!
|
||
- [`Hydra`](https://github.com/facebookresearch/hydra) is a framework for elegantly configuring complex applications.
|
||
- [`FastAPI`](https://github.com/tiangolo/fastapi) is a type-driven asynchronous web framework.
|
||
|
||
Articles:
|
||
|
||
- [Open Source Guides](https://opensource.guide/).
|
||
- [A handy guide to financial support for open source](https://github.com/nayafia/lemonade-stand)
|
||
- [GitHub Actions Documentation](https://help.github.com/en/actions).
|
||
- Maybe you would like to add [gitmoji](https://gitmoji.carloscuesta.me/) to commit names. This is really funny. 😄
|
||
|
||
## 🚀 Features
|
||
|
||
### Development features
|
||
|
||
- Supports for `Python 3.11` and higher.
|
||
- [`Poetry`](https://python-poetry.org/) as the dependencies manager. See configuration in [`pyproject.toml`](https://github.com/internet-ml/internet-ml/blob/master/pyproject.toml) and [`setup.cfg`](https://github.com/internet-ml/internet-ml/blob/master/setup.cfg).
|
||
- Automatic codestyle with [`black`](https://github.com/psf/black), [`isort`](https://github.com/timothycrosley/isort) and [`pyupgrade`](https://github.com/asottile/pyupgrade).
|
||
- Ready-to-use [`pre-commit`](https://pre-commit.com/) hooks with code-formatting.
|
||
- Type checks with [`mypy`](https://mypy.readthedocs.io); docstring checks with [`darglint`](https://github.com/terrencepreilly/darglint); security checks with [`safety`](https://github.com/pyupio/safety) and [`bandit`](https://github.com/PyCQA/bandit)
|
||
- Testing with [`pytest`](https://docs.pytest.org/en/latest/).
|
||
- Ready-to-use [`.editorconfig`](https://github.com/internet-ml/internet-ml/blob/master/.editorconfig), [`.dockerignore`](https://github.com/internet-ml/internet-ml/blob/master/.dockerignore), and [`.gitignore`](https://github.com/internet-ml/internet-ml/blob/master/.gitignore). You don't have to worry about those things.
|
||
|
||
### Deployment features
|
||
|
||
- `GitHub` integration: issue and pr templates.
|
||
- `Github Actions` with predefined [build workflow](https://github.com/internet-ml/internet-ml/blob/master/.github/workflows/build.yml) as the default CI/CD.
|
||
- Everything is already set up for security checks, codestyle checks, code formatting, testing, linting, docker builds, etc with [`Makefile`](https://github.com/internet-ml/internet-ml/blob/master/Makefile#L89). More details in [makefile-usage](#makefile-usage).
|
||
- [Dockerfile](https://github.com/internet-ml/internet-ml/blob/master/docker/Dockerfile) for your package.
|
||
- Always up-to-date dependencies with [`@dependabot`](https://dependabot.com/). You will only [enable it](https://docs.github.com/en/github/administering-a-repository/enabling-and-disabling-version-updates#enabling-github-dependabot-version-updates).
|
||
- Automatic drafts of new releases with [`Release Drafter`](https://github.com/marketplace/actions/release-drafter). You may see the list of labels in [`release-drafter.yml`](https://github.com/internet-ml/internet-ml/blob/master/.github/release-drafter.yml). Works perfectly with [Semantic Versions](https://semver.org/) specification.
|
||
|
||
### Open source community features
|
||
|
||
- Ready-to-use [Pull Requests templates](https://github.com/internet-ml/internet-ml/blob/master/.github/PULL_REQUEST_TEMPLATE.md) and several [Issue templates](https://github.com/internet-ml/internet-ml/tree/master/.github/ISSUE_TEMPLATE).
|
||
- Files such as: `LICENSE`, `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, and `SECURITY.md` are generated automatically.
|
||
- [`Stale bot`](https://github.com/apps/stale) that closes abandoned issues after a period of inactivity. (You will only [need to setup free plan](https://github.com/marketplace/stale)). Configuration is [here](https://github.com/internet-ml/internet-ml/blob/master/.github/.stale.yml).
|
||
- [Semantic Versions](https://semver.org/) specification with [`Release Drafter`](https://github.com/marketplace/actions/release-drafter).
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
pip install -U internet-ml
|
||
```
|
||
|
||
or install with `Poetry`
|
||
|
||
```bash
|
||
poetry add internet-ml
|
||
```
|
||
|
||
Then you can run
|
||
|
||
```bash
|
||
internet-ml --help
|
||
```
|
||
|
||
or with `Poetry`:
|
||
|
||
```bash
|
||
poetry run internet-ml --help
|
||
```
|
||
|
||
### Makefile usage
|
||
|
||
[`Makefile`](https://github.com/internet-ml/internet-ml/blob/master/Makefile) contains a lot of functions for faster development.
|
||
|
||
<details>
|
||
<summary>1. Download and remove Poetry</summary>
|
||
<p>
|
||
|
||
To download and install Poetry run:
|
||
|
||
```bash
|
||
make poetry-download
|
||
```
|
||
|
||
To uninstall
|
||
|
||
```bash
|
||
make poetry-remove
|
||
```
|
||
|
||
</p>
|
||
</details>
|
||
|
||
<details>
|
||
<summary>2. Install all dependencies and pre-commit hooks</summary>
|
||
<p>
|
||
|
||
Install requirements:
|
||
|
||
```bash
|
||
make install
|
||
```
|
||
|
||
Pre-commit hooks coulb be installed after `git init` via
|
||
|
||
```bash
|
||
make pre-commit-install
|
||
```
|
||
|
||
</p>
|
||
</details>
|
||
|
||
<details>
|
||
<summary>3. Codestyle</summary>
|
||
<p>
|
||
|
||
Automatic formatting uses `pyupgrade`, `isort` and `black`.
|
||
|
||
```bash
|
||
make codestyle
|
||
|
||
# or use synonym
|
||
make formatting
|
||
```
|
||
|
||
Codestyle checks only, without rewriting files:
|
||
|
||
```bash
|
||
make check-codestyle
|
||
```
|
||
|
||
> Note: `check-codestyle` uses `isort`, `black` and `darglint` library
|
||
|
||
Update all dev libraries to the latest version using one comand
|
||
|
||
```bash
|
||
make update-dev-deps
|
||
```
|
||
|
||
</p>
|
||
</details>
|
||
|
||
<details>
|
||
<summary>4. Code security</summary>
|
||
<p>
|
||
|
||
```bash
|
||
make check-safety
|
||
```
|
||
|
||
This command launches `Poetry` integrity checks as well as identifies security issues with `Safety` and `Bandit`.
|
||
|
||
```bash
|
||
make check-safety
|
||
```
|
||
|
||
</p>
|
||
</details>
|
||
|
||
<details>
|
||
<summary>5. Type checks</summary>
|
||
<p>
|
||
|
||
Run `mypy` static type checker
|
||
|
||
```bash
|
||
make mypy
|
||
```
|
||
|
||
</p>
|
||
</details>
|
||
|
||
<details>
|
||
<summary>6. Tests with coverage badges</summary>
|
||
<p>
|
||
|
||
Run `pytest`
|
||
|
||
```bash
|
||
make test
|
||
```
|
||
|
||
</p>
|
||
</details>
|
||
|
||
<details>
|
||
<summary>7. All linters</summary>
|
||
<p>
|
||
|
||
Of course there is a command to ~~rule~~ run all linters in one:
|
||
|
||
```bash
|
||
make lint
|
||
```
|
||
|
||
the same as:
|
||
|
||
```bash
|
||
make test && make check-codestyle && make mypy && make check-safety
|
||
```
|
||
|
||
</p>
|
||
</details>
|
||
|
||
<details>
|
||
<summary>8. Docker</summary>
|
||
<p>
|
||
|
||
```bash
|
||
make docker-build
|
||
```
|
||
|
||
which is equivalent to:
|
||
|
||
```bash
|
||
make docker-build VERSION=latest
|
||
```
|
||
|
||
Remove docker image with
|
||
|
||
```bash
|
||
make docker-remove
|
||
```
|
||
|
||
More information [about docker](https://github.com/internet-ml/internet-ml/tree/master/docker).
|
||
|
||
</p>
|
||
</details>
|
||
|
||
<details>
|
||
<summary>9. Cleanup</summary>
|
||
<p>
|
||
Delete pycache files
|
||
|
||
```bash
|
||
make pycache-remove
|
||
```
|
||
|
||
Remove package build
|
||
|
||
```bash
|
||
make build-remove
|
||
```
|
||
|
||
Delete .DS_STORE files
|
||
|
||
```bash
|
||
make dsstore-remove
|
||
```
|
||
|
||
Remove .mypycache
|
||
|
||
```bash
|
||
make mypycache-remove
|
||
```
|
||
|
||
Or to remove all above run:
|
||
|
||
```bash
|
||
make cleanup
|
||
```
|
||
|
||
</p>
|
||
</details>
|
||
|
||
## 📈 Releases
|
||
|
||
You can see the list of available releases on the [GitHub Releases](https://github.com/internet-ml/internet-ml/releases) page.
|
||
|
||
We follow [Semantic Versions](https://semver.org/) specification.
|
||
|
||
We use [`Release Drafter`](https://github.com/marketplace/actions/release-drafter). As pull requests are merged, a draft release is kept up-to-date listing the changes, ready to publish when you’re ready. With the categories option, you can categorize pull requests in release notes using labels.
|
||
|
||
### List of labels and corresponding titles
|
||
|
||
| **Label** | **Title in Releases** |
|
||
| :-----------------------------------: | :---------------------: |
|
||
| `enhancement`, `feature` | 🚀 Features |
|
||
| `bug`, `refactoring`, `bugfix`, `fix` | 🔧 Fixes & Refactoring |
|
||
| `build`, `ci`, `testing` | 📦 Build System & CI/CD |
|
||
| `breaking` | 💥 Breaking Changes |
|
||
| `documentation` | 📝 Documentation |
|
||
| `dependencies` | ⬆️ Dependencies updates |
|
||
|
||
You can update it in [`release-drafter.yml`](https://github.com/internet-ml/internet-ml/blob/master/.github/release-drafter.yml).
|
||
|
||
GitHub creates the `bug`, `enhancement`, and `documentation` labels for you. Dependabot creates the `dependencies` label. Create the remaining labels on the Issues tab of your GitHub repository, when you need them.
|
||
|
||
## 🛡 License
|
||
|
||
[](https://github.com/internet-ml/internet-ml/blob/master/LICENSE)
|
||
|
||
This project is licensed under the terms of the `GNU GPL v3.0` license. See [LICENSE](https://github.com/internet-ml/internet-ml/blob/master/LICENSE) for more details.
|
||
|
||
## 📃 Citation
|
||
|
||
```bibtex
|
||
@misc{internet-ml,
|
||
author = {Thamognya Kodi},
|
||
title = {Internet-ML: Allowing ML to connect to the internet},
|
||
year = {2022},
|
||
publisher = {GitHub},
|
||
journal = {GitHub repository},
|
||
howpublished = {\url{https://github.com/thamognya/internet-ml}}
|
||
}
|
||
```
|