[Bug]: Embedding服务无法使用，索引越界，导致出现error code:507018，无法自愈-text-embeddings-inference-AtomGit

已关闭

[Bug]: Embedding服务无法使用，索引越界，导致出现error code:507018，无法自愈 #129

fjigkf创建于 8月6日关闭于 8月11日

fjigkf

8月6日创建

在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

⚠️ 安全信息提醒：请仔细检查提供的文本内容，确保其不包含敏感数据信息，包括但不限于：

API 令牌或密钥
密码或身份验证凭证
私有网址或接口地址
个人或机密数据
...

在分享配置信息或代码示例时，请将敏感信息脱敏处理，或使用 <TOKEN> 等占位符替代原有内容。

环境信息

- 操作系统
EulerOS2.0 (SP12)
- 昇腾硬件信息
Atlas 300I Duo
- CANN软件版本
25.5.1
- 安装的对应软件版本
镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mis-tei:7.3.0-300I-Duo-aarch64

🐛 问题描述

报错日志

2026-08-05T10:16:27.277424Z ERROR python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:40: Method Embed encountered an error.
Traceback (most recent call last):
  File "/usr/local/bin/python-text-embeddings-server", line 7, in <module>
    sys.exit(app())
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 311, in __call__
    return get_command(self)(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 716, in main
    return _main(
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/cli.py", line 51, in serve
    server.serve(model_path, dtype, uds_path, pool)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 118, in serve
    asyncio.run(serve_inner(model_path, dtype))
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 637, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 604, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/grpc_interceptor/server.py", line 159, in invoke_intercept_method
    return await self.intercept(
> File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/utils/interceptor.py", line 21, in intercept
    return await response
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 120, in _unary_interceptor
    raise error
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 111, in _unary_interceptor
    return await behavior(request_or_iterator, context)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 39, in Embed
    embeddings = self.model.embed(batch)
  File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/models/default_model.py", line 91, in embed
    return self._process_default(batch, kwargs)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/models/default_model.py", line 122, in _process_default
    output = self.model(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/Ascend/mxRag/ops/transformer_adapter/modeling_bert_adapter.py", line 103, in forward_boost
    hidden_states = self.execute_ascend_operator_boost(input_ids,
  File "/usr/local/Ascend/mxRag/ops/transformer_adapter/common_adapater.py", line 123, in execute_ascend_operator_boost
    acl_hidden_state = acl_model_out[0]
IndexError: list index out of range

2026-08-05T10:16:27.277766Z ERROR backend_task:embed:embed: backend_grpc_client: backends/grpc-client/src/lib.rs:25: Server error: list index out of range
2026-08-05T10:16:27.278082Z ERROR embed:embed_pooled{truncate=false truncation_direction=Right prompt_name=None normalize=true dimensions=None}: text_embeddings_core::infer: core/src/infer.rs:384: Server error: list index out of range
2026-08-05T10:16:27.278140Z ERROR embed:embed_pooled{truncate=false truncation_direction=Right prompt_name=None normalize=true dimensions=None}: text_embeddings_core::infer: core/src/infer.rs:384: Server error: list index out of range
2026-08-05T10:16:27.278162Z ERROR embed:embed_pooled{truncate=false truncation_direction=Right prompt_name=None normalize=true dimensions=None}: text_embeddings_core::infer: core/src/infer.rs:384: Server error: list index out of range
2026-08-05T10:16:27.278183Z ERROR embed:embed_pooled{truncate=false truncation_direction=Right prompt_name=None normalize=true dimensions=None}: text_embeddings_core::infer: core/src/infer.rs:384: Server error: list index out of range
2026-08-05T10:16:27.320228Z ERROR python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:40: Method Embed encountered an error.
Traceback (most recent call last):
  File "/usr/local/bin/python-text-embeddings-server", line 7, in <module>
    sys.exit(app())
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 311, in __call__
    return get_command(self)(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 716, in main
    return _main(
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/cli.py", line 51, in serve
    server.serve(model_path, dtype, uds_path, pool)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 118, in serve
    asyncio.run(serve_inner(model_path, dtype))
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 637, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 604, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/grpc_interceptor/server.py", line 159, in invoke_intercept_method
    return await self.intercept(
> File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/utils/interceptor.py", line 21, in intercept
    return await response
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 120, in _unary_interceptor
    raise error
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 111, in _unary_interceptor
    return await behavior(request_or_iterator, context)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 35, in Embed
    batch = self.model.batch_type.from_pb(
  File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/models/types.py", line 79, in from_pb
    all_tensors = all_tensors.to(device)
RuntimeError: ACL stream synchronize failed, error code:507018

2026-08-05T10:16:27.320456Z ERROR backend_task:embed:embed: backend_grpc_client: backends/grpc-client/src/lib.rs:25: Server error: ACL stream synchronize failed, error code:507018
2026-08-05T10:16:27.320740Z ERROR embed:embed_pooled{truncate=false truncation_direction=Right prompt_name=None normalize=true dimensions=None}: text_embeddings_core::infer: core/src/infer.rs:384: Server error: ACL stream synchronize failed, error code:507018
2026-08-05T10:16:27.320790Z ERROR embed:embed_pooled{truncate=false truncation_direction=Right prompt_name=None normalize=true dimensions=None}: text_embeddings_core::infer: core/src/infer.rs:384: Server error: ACL stream synchronize failed, error code:507018
2026-08-05T10:16:27.320810Z ERROR embed:embed_pooled{truncate=false truncation_direction=Right prompt_name=None normalize=true dimensions=None}: text_embeddings_core::infer: core/src/infer.rs:384: Server error: ACL stream synchronize failed, error code:507018
2026-08-06T01:23:17.784814Z  INFO batching_task:next_batch: text_embeddings_core::queue: core/src/queue.rs:188: inference batch size is:2
2026-08-06T01:23:17.794054Z ERROR python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:40: Method Embed encountered an error.
Traceback (most recent call last):
  File "/usr/local/bin/python-text-embeddings-server", line 7, in <module>
    sys.exit(app())
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 311, in __call__
    return get_command(self)(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 716, in main
    return _main(
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/cli.py", line 51, in serve
    server.serve(model_path, dtype, uds_path, pool)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 118, in serve
    asyncio.run(serve_inner(model_path, dtype))
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 637, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 604, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/grpc_interceptor/server.py", line 159, in invoke_intercept_method
    return await self.intercept(
> File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/utils/interceptor.py", line 21, in intercept
    return await response
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 120, in _unary_interceptor
    raise error
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 111, in _unary_interceptor
    return await behavior(request_or_iterator, context)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/server.py", line 35, in Embed
    batch = self.model.batch_type.from_pb(
  File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
  File "/home/HwHiAiUser/text-embeddings-inference/backends/python/server/text_embeddings_server/models/types.py", line 79, in from_pb
    all_tensors = all_tensors.to(device)
RuntimeError: ACL stream synchronize failed, error code:507018

欢迎加入社区，感谢您对社区的贡献 🎉!