AAkhilesh Sharmabuild: commit Cargo.lock for reproducible builds

f7aafd9a创建于 9月10日历史提交

文件	最后提交记录	最后更新时间
migrations	new monthly usage	1 年前
src	chore: fixed feature flags in tests	1 年前
.env.example	feat: allow users to choose an LLM model to use for `segment_processing` by setting the `llm_processing.model_id` param in the POST and PATCH request. The available models can be configured using the `models.yaml` file. (#437) * exposed llm model and url to requester * llm processing custom models * llm model expose * compose * Update compose-cpu.yaml * Update llm_config.rs * addressed comments * addressed all comments except for .env and yaml merging in llmconfig - unsure about llm_ocr usage * fixed throttling and llm service * LLM working successfully * updated configuration to switch error_handling and llm_processing to have defaults instead of optional * added checks to create and upload routes to throw 400 erros if model_ids not in config * added route for llm models * updated llm processing deserializing to automatically replace defaults with actual model names and added check for outdated models in the update route * llm throttle is now model based rather than domain based * improved LLM fallbacks * all tests passed * formatted code * updated docker compose file replicas * updated models.yaml examples * added config map for models.yaml * updated README * removed example models.yaml from core --------- Co-authored-by: Akhilesh Sharma <sharmaninenine@gmail.com>	1 年前
.gitignore	build: commit Cargo.lock for reproducible builds The .gitignore rule 'Ignore Cargo.lock if it is a library' was applied to binary crates, so neither server had a committed lockfile. Dependency versions therefore resolved fresh on every build. Commit the lockfiles and remove the ignore rule so builds are reproducible and the resolved dependency set is recorded.	1 个月前
CHANGELOG.md	chore: release main (#569) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	1 年前
Cargo.lock	build: commit Cargo.lock for reproducible builds The .gitignore rule 'Ignore Cargo.lock if it is a library' was applied to binary crates, so neither server had a committed lockfile. Dependency versions therefore resolved fresh on every build. Commit the lockfiles and remove the ignore rule so builds are reproducible and the resolved dependency set is recorded.	1 个月前
Cargo.toml	chore: release main (#569) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	1 年前
README.md	feat: added Open telemetry support for better analytics (#504) * set up otel collector and opentelemetry in server * added tracing to create task * removed api key from trace * updated crates * updated create_task span to use the same trace as actix * added user telemetry in middleware * propagated context to task worker * added spans for pipeline steps * added better messaging during file conversion * removed macro and updated error handling * added retry tracing * updated step span to have one for each retry * added llm level spans * fixed step level context issue * traces till open ai request added, need to fix parent-child relationships * fixed parent-child relationships * updated llm error * added better error messages for llm errors * updated attributes for better analytics * updated README for otel collector * fixed clippy warnings	1 年前
build.rs	feat: allow users to choose an LLM model to use for `segment_processing` by setting the `llm_processing.model_id` param in the POST and PATCH request. The available models can be configured using the `models.yaml` file. (#437) * exposed llm model and url to requester * llm processing custom models * llm model expose * compose * Update compose-cpu.yaml * Update llm_config.rs * addressed comments * addressed all comments except for .env and yaml merging in llmconfig - unsure about llm_ocr usage * fixed throttling and llm service * LLM working successfully * updated configuration to switch error_handling and llm_processing to have defaults instead of optional * added checks to create and upload routes to throw 400 erros if model_ids not in config * added route for llm models * updated llm processing deserializing to automatically replace defaults with actual model names and added check for outdated models in the update route * llm throttle is now model based rather than domain based * improved LLM fallbacks * all tests passed * formatted code * updated docker compose file replicas * updated models.yaml examples * added config map for models.yaml * updated README * removed example models.yaml from core --------- Co-authored-by: Akhilesh Sharma <sharmaninenine@gmail.com>	1 年前
diesel.toml	merged	1 年前

This is holds the core of the chunkr api. The API consists of multiple services that work together to extract structured data from documents.

Development

Setup

# Install Rust and Cargo
curl https://sh.rustup.rs -sSf | sh

# Source the cargo environment
source $HOME/.cargo/env

# Copy the example env file
cp .env.example .env

Running the Services

Rust services

# Start the server
cargo run

# Start the workers
## Each worker is 1 task
cargo run --bin task

Other Services

To run the other services it is recommended to use the docker compose file in the root of the repo. For Docker Compose setup and usage instructions, please refer to Quick Start with Docker Compose.

Set replicas to 0 for services you don't want to run/aren't actively being worked on.