Overview

What is MLSync?

MLSync is a Python library that acts as a bridge between your ML workflow and your project planning and management tools.

Installation

pip install mlsync

Why MLSync?

Developing ML projects is a lot of fun, but they are also hard to plan and manage. While the ML community has built several tools for developers to better track and visualize their ML workflow data, there is a disconnect between ML workflow data and the tools that are used for project management. MLSync is designed to bridge this gap.

How Does it Work?

There are four main aspects of MLSync:

MLSync interfaces with modern ML experiment tracking tools such as MLflow and imports the raw data.
Raw data from ML experiment tracking tools are converted to MLSync internal data format (user-defined) and stored in a database.
MLSync engine processes this raw data and generates consolidated insights for your project.
The insights are then converted to suitable formats and sent to your project planning and management tools such as Notion.

We are actively building MLSync with the vision to become a one-stop standard interface to map data from ML experiments to project management tools. The above figure shows the high-level architecture of MLSync. All the functionality is not yet available; please refer to the Roadmap for the current status. If you would like to contribute to MLSync, please refer to the Contributing section.

Example

In this example, we will sync your machine learning experiments to Notion in three simple steps!

1. Install MLSync

pip install mlsync

2. Setup the Example

git clone https://github.com/paletteml/mlsync.git: Checkout the MLSync repository.
cd mlsync/examples/mlflow-notion/: Change directory to the example directory
pip install -r requirements.txt : Install the requirements for this example.
- Note that the above step installs Pytorch. If you run into issues, please refer to the Pytorch documentation for more information.
Run example training using python mlflow_pytorch.py --run-name <name>. Make sure it runs (Need not complete the run).

3. Notion Setup

Let us now link Notion to MLSync. This is required only for the first time you run MLSync.

Create a new integration to Notion.
1. Visit notion.so/my-integrations
2. Click the + New Integration button.
3. Name it as MLSync and hit submit.
4. Copy your “Internal Integration Token” from your Notion integration page.
5. Open the .env file in your path and update the Notion token.
  - NOTION_TOKEN=secret_0000000000000000000000000000000000000000000
Create a new page in Notion. This will serve as the root page for your MLFlow runs.
1. Let us name the page as Demo.
2. Click the Share button on the top right corner of the page.
3. Click the Invite button and then choose MLSync integration.

All Done

You are now all set! Now let us sync your MLFlow runs to Notion.

mlsync --config config.yaml

{% note %}

Note: First time you run, you will be prompted to choose a page to sync to. From the options, choose the page you created in the previous step (Demo).

{% endnote %}

That’s it! You can now view your MLFlow runs in Notion. As long as mlsync is running in the background, all your future experiments and runs in this directory should appear in the selected Notion page.

Troubleshooting

If you are getting an error related to the NOTION_TOKEN not being found, you can pass the --notion-token flag to mlsync to specify the token.
If you are having trouble with MNIST dataflow download, you can try to download the data manually from here.
Please contact us for any other issues.

Please raise an issue, or reach out if you have any other errors.

Advanced

You can override the Notion page id, token, and other configurations by either modifying the config.yaml file or by passing the arguments to the mlsync command. Run mlsync --help to see the available arguments.
Custom Report Formats: mlsync allows you to customize the report much further. You can customize the report by adding your own format.yaml file. Read documentation here to learn more.
Custom Refresh Rates: You can control the refresh rate of the report by setting the refresh_rate field in the configuration file.
Restarting mlsync: You can restart mlsync any time without losing earlier runs.

Enjoy! If you have any further questions, please contact us.

Roadmap

We want to support different training environments and different productivity tools.

Productivity Tools
1. Notion: Supported
2. Trello: Planned
3. Confluence: In progress
4. Jira: Planned
Monitoring Frameworks
1. MLFlow: Supported
2. TensorBoard: In progress
3. ClearML: Planned
Programmatic API
1. Planned

Do you have other tools/frameworks you would like to see supported? Let us know!

Contributing

We welcome contributions from the community. Please feel free to open an issue or pull request. Or, if you are interested in working closely with us, please contact us directly. We will be happy to talk to you!