stumpy:高效计算矩阵轮廓,助力时间序列数据挖掘

STUMPY is a powerful and scalable Python library for modern time series analysis

分支2Tags30
文件最后提交记录最后更新时间
1 个月前
27 天前
3 年前
6 年前
1 个月前
15 天前
7 个月前
2 年前
1 年前
6 年前
7 个月前
1 年前
6 个月前
4 年前
7 年前
1 年前
7 年前
2 年前
7 个月前
2 年前
2 年前
26 天前
7 个月前
3 个月前
1 年前
5 年前
4 年前
5 年前
6 个月前
1 个月前
6 年前
9 个月前
2 年前
1 个月前
1 个月前
3 年前

|PyPI 版本| |Conda Forge 版本| |PyPI 下载量| |许可证| |测试状态| |代码覆盖率|

|RTD 状态| |Binder| |JOSS| |NumFOCUS|

.. |PyPI Version| image:: https://img.shields.io/pypi/v/stumpy.svg :target: https://pypi.org/project/stumpy/ :alt: PyPI 版本 .. |Conda Forge Version| image:: https://anaconda.org/conda-forge/stumpy/badges/version.svg :target: https://anaconda.org/conda-forge/stumpy :alt: Conda-Forge 版本 .. |PyPI Downloads| image:: https://static.pepy.tech/badge/stumpy/month :target: https://pepy.tech/project/stumpy :alt: PyPI 下载量 .. |License| image:: https://img.shields.io/pypi/l/stumpy.svg :target: https://github.com/stumpy-dev/stumpy/blob/main/LICENSE.txt :alt: 许可证 .. |Test Status| image:: https://github.com/stumpy-dev/stumpy/workflows/Tests/badge.svg :target: https://github.com/stumpy-dev/stumpy/actions?query=workflow%3ATests+branch%3Amain :alt: 测试状态 .. |Code Coverage| image:: https://img.shields.io/badge/Coverage-100%25-green :alt: 代码覆盖率 .. |RTD Status| image:: https://readthedocs.org/projects/stumpy/badge/?version=latest :target: https://stumpy.readthedocs.io/ :alt: ReadTheDocs 状态 .. |Binder| image:: https://mybinder.org/badge_logo.svg :target: https://mybinder.org/v2/gh/stumpy-dev/stumpy/main?filepath=notebooks :alt: Binder .. |JOSS| image:: http://joss.theoj.org/papers/10.21105/joss.01504/status.svg :target: https://doi.org/10.21105/joss.01504 :alt: JOSS .. |DOI| image:: https://zenodo.org/badge/184809315.svg :target: https://zenodo.org/badge/latestdoi/184809315 :alt: DOI .. |NumFOCUS| image:: https://img.shields.io/badge/NumFOCUS-Affiliated Project-orange.svg?style=flat&colorA=E1523D&colorB=007D8A :target: https://numfocus.org/sponsored-projects/affiliated-projects :alt: NumFOCUS 附属项目 .. |Twitter| image:: https://img.shields.io/twitter/follow/stumpy_dev.svg?style=social :target: https://twitter.com/stumpy_dev :alt: Twitter

|

.. image:: https://raw.githubusercontent.com/stumpy-dev/stumpy/main/docs/images/stumpy_logo_small.png :target: https://github.com/stumpy-dev/stumpy :alt: STUMPY 徽标

====== STUMPY

STUMPY 是一个功能强大且可扩展的 Python 库,能高效计算一种名为“矩阵轮廓(matrix profile)”的指标。通俗来讲,它的作用就是:针对时间序列中每一个(绿色的)子序列,自动找出与其最相似的近邻子序列(灰色的)。详情可参考:matrix profile

.. image:: https://github.com/stumpy-dev/stumpy/blob/main/docs/images/stumpy_demo.gif?raw=true :alt: STUMPY Animated GIF

关键在于,一旦计算出矩阵轮廓(如上图中间面板所示),它就能应用于多种时间序列数据挖掘任务,例如:

  • 模式/基元(motif)发现(即在较长时间序列中找出近似重复的子序列)
  • 异常/新奇值(discord)发现
  • 形状特征(shapelet)发现
  • 语义分割
  • 流数据(在线数据)处理
  • 快速近似矩阵轮廓计算
  • 时间序列链(按时间顺序排列的子序列模式集合)
  • 长时序数据摘要片段(snippets)生成
  • 全景矩阵轮廓(pan matrix profiles)用于选择最佳子序列窗口大小
  • 以及更多...

无论您是学术界人士、数据科学家、软件开发工程师,还是时间序列爱好者,STUMPY 的安装都十分简便。我们的目标是帮助您更快地从时间序列数据中洞察有价值的信息。更多详情,请参见 documentation


如何使用 STUMPY

完整的可用函数列表,请参见我们的 API 文档。更全面的示例使用场景,请查阅我们内容丰富的 教程。下文将提供一些代码片段,快速演示 STUMPY 的使用方法。

使用 STUMP 处理典型一维时间序列数据:

.. code:: python

import stumpy import numpy as np

if __name__ == "__main__":
    your_time_series = np.random.rand(10000)
    window_size = 50  # 一个模式中可能包含的数据点数量(近似值)

    matrix_profile = stumpy.stump(your_time_series, m=window_size)

通过 Dask Distributed 使用 STUMPED 进行分布式一维时间序列数据处理:

.. code:: python

import stumpy import numpy as np from dask.distributed import Client

if __name__ == "__main__":
    with Client() as dask_client:
        your_time_series = np.random.rand(10000)
        window_size = 50  # 一个模式中可能包含的数据点数量(近似值)

        matrix_profile = stumpy.stumped(dask_client, your_time_series, m=window_size)

使用 GPU-STUMP 进行 GPU 加速的一维时间序列数据处理:

.. code:: python

import stumpy import numpy as np from numba import cuda

if __name__ == "__main__":
    your_time_series = np.random.rand(10000)
    window_size = 50  # 一个模式中可能包含的数据点数量(近似值)
    all_gpu_devices = [device.id for device in cuda.list_devices()]  # 获取所有可用 GPU 设备列表

    matrix_profile = stumpy.gpu_stump(your_time_series, m=window_size, device_id=all_gpu_devices)

使用 MSTUMP 处理多维时间序列数据:

.. code:: python

import stumpy import numpy as np

if __name__ == "__main__":
    your_time_series = np.random.rand(3, 1000)  # 每一行代表一个不同维度的数据,每一列代表同一时刻不同维度的数据
    window_size = 50  # 一个模式中可能包含的数据点数量(近似值)

    matrix_profile, matrix_profile_indices = stumpy.mstump(your_time_series, m=window_size)

使用 Dask Distributed 和 MSTUMPED 进行分布式多维时间序列数据分析:

.. code:: python

import stumpy import numpy as np from dask.distributed import Client

if __name__ == "__main__":
    with Client() as dask_client:
        your_time_series = np.random.rand(3, 1000)   # 每一行代表一个不同维度的数据,每一列代表同一时刻不同维度的数据
        window_size = 50  # 一个模式中可能包含的数据点数量(近似值)

        matrix_profile, matrix_profile_indices = stumpy.mstumped(dask_client, your_time_series, m=window_size)

使用 锚定时间序列链(Anchored Time Series Chains, ATSC) 进行时间序列链分析:

.. code:: python

import stumpy import numpy as np

if __name__ == "__main__":
    your_time_series = np.random.rand(10000)
    window_size = 50  # 一个模式中可能包含的数据点数量(近似值)
    
    matrix_profile = stumpy.stump(your_time_series, m=window_size)

    left_matrix_profile_index = matrix_profile[:, 2]
    right_matrix_profile_index = matrix_profile[:, 3]
    idx = 10  # 要获取其锚定时间序列链的子序列索引

    anchored_chain = stumpy.atsc(left_matrix_profile_index, right_matrix_profile_index, idx)

    all_chain_set, longest_unanchored_chain = stumpy.allc(left_matrix_profile_index, right_matrix_profile_index)

使用 快速低成本单能语义分割(Fast Low-cost Unipotent Semantic Segmentation, FLUSS) 进行语义分割:

.. code:: python

import stumpy import numpy as np

if __name__ == "__main__":
    your_time_series = np.random.rand(10000)
    window_size = 50  # 一个模式中可能包含的数据点数量(近似值)

    matrix_profile = stumpy.stump(your_time_series, m=window_size)

    subseq_len = 50
    correct_arc_curve, regime_locations = stumpy.fluss(matrix_profile[:, 1], 
                                                    L=subseq_len, 
                                                    n_regimes=2, 
                                                    excl_factor=1
                                                    )

依赖项

支持的 Python 和 NumPy 版本根据 NEP 29 弃用政策 确定。


获取方式

conda:

.. code:: bash

conda install -c conda-forge stumpy

pip:

.. code:: bash

python -m pip install stumpy

pixi:

.. code:: bash

pixi add stumpy

uv:

.. code:: bash

uv add stumpy

要从源代码安装 stumpy,请参阅 文档 中的说明。


文档

为了充分理解和欣赏底层算法及应用,您必须阅读原始 publications_。有关如何使用 STUMPY 的更详细示例,请查阅最新的 文档 或浏览我们的 实践教程


性能

我们使用 Numba JIT 编译版本的代码,在不同长度的随机生成时间序列数据(即 np.random.rand(n))以及不同的 CPU 和 GPU 硬件资源 上测试了计算精确矩阵轮廓的性能。

.. image:: https://raw.githubusercontent.com/stumpy-dev/stumpy/main/docs/images/performance.png :alt: STUMPY 性能图

原始结果以“时:分:秒.毫秒”格式显示在下表中,窗口大小固定为 m = 50。请注意,报告的运行时间包括将数据从主机传输到所有 GPU 设备所花费的时间。您可能需要向右滚动表格才能查看所有运行时间。

+----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | i | n = 2\ :sup:i | GPU-STOMP | STUMP.2 | STUMP.16 | STUMPED.128 | STUMPED.256 | GPU-STUMP.1 | GPU-STUMP.2 | GPU-STUMP.DGX1 | GPU-STUMP.DGX2 | +==========+===================+==============+=============+=============+=============+=============+=============+=============+================+================+ | 6 | 64 | 00:00:10.00 | 00:00:00.00 | 00:00:00.00 | 00:00:05.77 | 00:00:06.08 | 00:00:00.03 | 00:00:01.63 | NaN | NaN | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 7 | 128 | 00:00:10.00 | 00:00:00.00 | 00:00:00.00 | 00:00:05.93 | 00:00:07.29 | 00:00:00.04 | 00:00:01.66 | NaN | NaN | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 8 | 256 | 00:00:10.00 | 00:00:00.00 | 00:00:00.01 | 00:00:05.95 | 00:00:07.59 | 00:00:00.08 | 00:00:01.69 | 00:00:06.68 | 00:00:25.68 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 9 | 512 | 00:00:10.00 | 00:00:00.00 | 00:00:00.02 | 00:00:05.97 | 00:00:07.47 | 00:00:00.13 | 00:00:01.66 | 00:00:06.59 | 00:00:27.66 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 10 | 1024 | 00:00:10.00 | 00:00:00.02 | 00:00:00.04 | 00:00:05.69 | 00:00:07.64 | 00:00:00.24 | 00:00:01.72 | 00:00:06.70 | 00:00:30.49 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 11 | 2048 | NaN | 00:00:00.05 | 00:00:00.09 | 00:00:05.60 | 00:00:07.83 | 00:00:00.53 | 00:00:01.88 | 00:00:06.87 | 00:00:31.09 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 12 | 4096 | NaN | 00:00:00.22 | 00:00:00.19 | 00:00:06.26 | 00:00:07.90 | 00:00:01.04 | 00:00:02.19 | 00:00:06.91 | 00:00:33.93 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 13 | 8192 | NaN | 00:00:00.50 | 00:00:00.41 | 00:00:06.29 | 00:00:07.73 | 00:00:01.97 | 00:00:02.49 | 00:00:06.61 | 00:00:33.81 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 14 | 16384 | NaN | 00:00:01.79 | 00:00:00.99 | 00:00:06.24 | 00:00:08.18 | 00:00:03.69 | 00:00:03.29 | 00:00:07.36 | 00:00:35.23 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 15 | 32768 | NaN | 00:00:06.17 | 00:00:02.39 | 00:00:06.48 | 00:00:08.29 | 00:00:07.45 | 00:00:04.93 | 00:00:07.02 | 00:00:36.09 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 16 | 65536 | NaN | 00:00:22.94 | 00:00:06.42 | 00:00:07.33 | 00:00:09.01 | 00:00:14.89 | 00:00:08.12 | 00:00:08.10 | 00:00:36.54 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 17 | 131072 | 00:00:10.00 | 00:01:29.27 | 00:00:19.52 | 00:00:09.75 | 00:00:10.53 | 00:00:29.97 | 00:00:15.42 | 00:00:09.45 | 00:00:37.33 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 18 | 262144 | 00:00:18.00 | 00:05:56.50 | 00:01:08.44 | 00:00:33.38 | 00:00:24.07 | 00:00:59.62 | 00:00:27.41 | 00:00:13.18 | 00:00:39.30 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 19 | 524288 | 00:00:46.00 | 00:25:34.58 | 00:03:56.82 | 00:01:35.27 | 00:03:43.66 | 00:01:56.67 | 00:00:54.05 | 00:00:19.65 | 00:00:41.45 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 20 | 1048576 | 00:02:30.00 | 01:51:13.43 | 00:19:54.75 | 00:04:37.15 | 00:03:01.16 | 00:05:06.48 | 00:02:24.73 | 00:00:32.95 | 00:00:46.14 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 21 | 2097152 | 00:09:15.00 | 09:25:47.64 | 03:05:07.64 | 00:13:36.51 | 00:08:47.47 | 00:20:27.94 | 00:09:41.43 | 00:01:06.51 | 00:01:02.67 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 22 | 4194304 | NaN | 36:12:23.74 | 10:37:51.21 | 00:55:44.43 | 00:32:06.70 | 01:21:12.33 | 00:38:30.86 | 00:04:03.26 | 00:02:23.47 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 23 | 8388608 | NaN | 143:16:09.94| 38:42:51.42 | 03:33:30.53 | 02:00:49.37 | 05:11:44.45 | 02:33:14.60 | 00:15:46.26 | 00:08:03.76 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 24 | 16777216 | NaN | NaN | NaN | 14:39:11.99 | 07:13:47.12 | 20:43:03.80 | 09:48:43.42 | 01:00:24.06 | 00:29:07.84 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | NaN | 17729800 | 09:16:12.00 | NaN | NaN | 15:31:31.75 | 07:18:42.54 | 23:09:22.43 | 10:54:08.64 | 01:07:35.39 | 00:32:51.55 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 25 | 33554432 | NaN | NaN | NaN | 56:03:46.81 | 26:27:41.29 | 83:29:21.06 | 39:17:43.82 | 03:59:32.79 | 01:54:56.52 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 26 | 67108864 | NaN | NaN | NaN | 211:17:37.60| 106:40:17.17| 328:58:04.68| 157:18:30.50| 15:42:15.94 | 07:18:52.91 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | NaN | 100000000 | 291:07:12.00 | NaN | NaN | NaN | 234:51:35.39| NaN | NaN | 35:03:44.61 | 16:22:40.81 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+ | 27 | 134217728 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 64:41:55.09 | 29:13:48.12 | +----------+-------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+----------------+----------------+

^^^^^^^^^^^^^^^^^^ 硬件资源 ^^^^^^^^^^^^^^^^^^

.. _hardware:

GPU-STOMP:这些结果复制自原始 Matrix Profile II 论文——NVIDIA Tesla K80(包含 2 块 GPU),用作性能基准进行比较。

STUMP.2:stumpy.stump 在单台服务器上使用总共 2 个 CPU(2 个 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz 处理器)通过 Numba 并行执行,不使用 Dask。

STUMP.16:stumpy.stump 在单台服务器上使用总共 16 个 CPU(16 个 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz 处理器)通过 Numba 并行执行,不使用 Dask。

STUMPED.128:stumpy.stumped 在 16 台服务器上使用总共 128 个 CPU(每台服务器 8 个 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz 处理器),通过 Numba 并行化,并通过 Dask Distributed 进行分布式处理。

STUMPED.256:stumpy.stumped 在 32 台服务器上使用总共 256 个 CPU(每台服务器 8 个 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz 处理器),通过 Numba 并行化,并通过 Dask Distributed 进行分布式处理。

GPU-STUMP.1:stumpy.gpu_stump 使用 1 块 NVIDIA GeForce GTX 1080 Ti GPU,每块 512 个线程,200W 功率限制,通过 Numba 编译为 CUDA,并通过 Python 多进程并行化。

GPU-STUMP.2:stumpy.gpu_stump 使用 2 块 NVIDIA GeForce GTX 1080 Ti GPU,每块 512 个线程,200W 功率限制,通过 Numba 编译为 CUDA,并通过 Python 多进程并行化。

GPU-STUMP.DGX1:stumpy.gpu_stump 使用 8 块 NVIDIA Tesla V100,每块 512 个线程,通过 Numba 编译为 CUDA,并通过 Python 多进程并行化。

GPU-STUMP.DGX2:stumpy.gpu_stump 使用 16 块 NVIDIA Tesla V100,每块 512 个线程,通过 Numba 编译为 CUDA,并通过 Python 多进程并行化。


运行测试

测试用例编写在 tests 目录下,使用 PyTest 进行处理,并且需要 coverage.py 来进行代码覆盖率分析。可以通过以下命令执行测试:

.. code:: bash

./test.sh


Python 版本

STUMPY 支持 Python 3.10+,由于使用了 Unicode 变量名/标识符,因此与 Python 2.x 不兼容。鉴于依赖较少,STUMPY 可能在较旧版本的 Python 上运行,但这超出了我们的支持范围,我们强烈建议您升级到最新版本的 Python。


获取帮助

首先,请查看 Github 上的 讨论区问题列表,看看您的问题是否已经在那里得到解答。如果没有找到解决方案,欢迎发起新的讨论或 issue,作者会尽量及时回复。


贡献代码

我们欢迎任何形式的 贡献!文档方面的帮助,尤其是扩展教程,总是受欢迎的。要贡献代码,请 Fork 本项目,进行修改,然后提交拉取请求。我们会尽力与您一起解决任何问题,并将您的代码合并到主分支。


引用

如果您在科学出版物中使用了本代码库并希望引用它,请使用 《开源软件期刊》上的文章

S.M. Law, (2019). STUMPY: A Powerful and Scalable Python Library for Time Series Data Mining. Journal of Open Source Software, 4(39), 1504.

.. code:: bibtex

@article{law2019stumpy, author = {Law, Sean M.}, title = {{STUMPY: A Powerful and Scalable Python Library for Time Series Data Mining}}, journal = {{The Journal of Open Source Software}}, volume = {4}, number = {39}, pages = {1504}, year = {2019} }


参考文献

.. _publications:

Yeh, Chin-Chia Michael, 等. (2016) 矩阵轮廓 I:时间序列的全对相似性连接:一种包含基序、异常值和形状特征的统一视角。ICDM:1317-1322. 链接

Zhu, Yan, 等. (2016) 矩阵轮廓 II:利用新算法和 GPU 突破亿级时间序列基序和连接的障碍。ICDM:739-748. 链接

Yeh, Chin-Chia Michael, 等. (2017) 矩阵轮廓 VI:有意义的多维基序发现。ICDM:565-574. 链接

Zhu, Yan, 等. (2017) 矩阵轮廓 VII:时间序列链:时间序列数据挖掘的新基元。ICDM:695-704. 链接

Gharghabi, Shaghayegh, 等. (2017) 矩阵轮廓 VIII:超人类性能水平的领域无关在线语义分割。ICDM:117-126. 链接

Zhu, Yan, 等. (2017) 利用新算法和 GPU 突破千万亿级成对比较障碍用于时间序列基序和连接。KAIS:203-236. 链接

Zhu, Yan, 等. (2018) 矩阵轮廓 XI:SCRIMP++:交互式速度的时间序列基序发现。ICDM:837-846. 链接

Yeh, Chin-Chia Michael, 等. (2018) 时间序列连接、基序、异常值和形状特征:利用矩阵轮廓的统一视角。Data Min Knowl Disc:83-123. 链接

Gharghabi, Shaghayegh, 等. (2018) "Matrix Profile XII: MPdist: A Novel Time Series Distance Measure to Allow Data Mining in More Challenging Scenarios." ICDM:965-970. 链接

Zimmerman, Zachary, 等. (2019) 矩阵轮廓 XIV:使用 GPU 扩展时间序列基序发现以突破每日千万亿次成对比较及以上。SoCC '19:74-86. 链接

Akbarinia, Reza, 和 Betrand Cloez. (2019) 使用不同距离函数的高效矩阵轮廓计算。arXiv:1901.05708. 链接

Kamgar, Kaveh, 等. (2019) 矩阵轮廓 XV:利用时间序列共识基序发现时间序列集合中的结构。ICDM:1156-1161. 链接


许可协议与商标

| STUMPY | 2019年版权所有,归TD Ameritrade所有。根据3条款BSD许可协议的条款发布。 | STUMPY是TD Ameritrade IP Company, Inc.的商标。保留所有权利。