| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
add anya, scripts | 3 年前 | |
[DRAFT] Zero123 Improvements (#216) Multiple improvements to Zero123 and other files: - Sets good params for using Zero123! Doesn't use depth or normal images - Changes background for Zero123 to non-trainable solid-color-background with white color - Changes optimizer to Adam - The normals are much smoother. Will affect details like thin fingers, etc. But works well overall, and is very conducive to Phase 2 using tetrahedra-sdf-grid - Changes zero123-prompt-processor to dummy-prompt-processor - Adds resolution_milestones to data/image.py : so the height and width can change with iters. For example in zero123.yaml: data: # threestudio/data/image.py -> SingleImageDataModuleConfig image_path: ./load/images/hamburger_rgba.png height: [128, 256, 512] width: [128, 256, 512] resolution_milestones: [200, 300] - Only loads depth and normal images if lambda_depth or lambda_depth_rel > 0, or lambda_normal > 0. Else doesn't load them, only loads the _rgba image. Also, resizes the images according to resolution_milestones - Adds random_aug to solid-colour-background : uses a random background colour with probability random_aug_prob - Sets bg_color as white, changed ambient_ratio from 0.1 + 0.9*rand to 0.5 + 0.5*rand : didn't notice any difference with this change GEOMETRY REFINEMENT using tetrahedra-sdf-grid - Fixes to dataloaders to be compatible with nvdiffrast - Adds scripts to refine Phase1 results using tetrahedra-sdf-grid : configs/zero123-geometry.yaml and configs/experimental/imagecondition_refine.yaml - Adds normal_consistency loss during refinement - Adds laplacian_smoothness loss during refinement : doesn't work yet, will fix later | 3 年前 | |
[DRAFT] Zero123 Improvements (#216) Multiple improvements to Zero123 and other files: - Sets good params for using Zero123! Doesn't use depth or normal images - Changes background for Zero123 to non-trainable solid-color-background with white color - Changes optimizer to Adam - The normals are much smoother. Will affect details like thin fingers, etc. But works well overall, and is very conducive to Phase 2 using tetrahedra-sdf-grid - Changes zero123-prompt-processor to dummy-prompt-processor - Adds resolution_milestones to data/image.py : so the height and width can change with iters. For example in zero123.yaml: data: # threestudio/data/image.py -> SingleImageDataModuleConfig image_path: ./load/images/hamburger_rgba.png height: [128, 256, 512] width: [128, 256, 512] resolution_milestones: [200, 300] - Only loads depth and normal images if lambda_depth or lambda_depth_rel > 0, or lambda_normal > 0. Else doesn't load them, only loads the _rgba image. Also, resizes the images according to resolution_milestones - Adds random_aug to solid-colour-background : uses a random background colour with probability random_aug_prob - Sets bg_color as white, changed ambient_ratio from 0.1 + 0.9*rand to 0.5 + 0.5*rand : didn't notice any difference with this change GEOMETRY REFINEMENT using tetrahedra-sdf-grid - Fixes to dataloaders to be compatible with nvdiffrast - Adds scripts to refine Phase1 results using tetrahedra-sdf-grid : configs/zero123-geometry.yaml and configs/experimental/imagecondition_refine.yaml - Adds normal_consistency loss during refinement - Adds laplacian_smoothness loss during refinement : doesn't work yet, will fix later | 3 年前 | |
[DRAFT] Zero123 Improvements (#216) Multiple improvements to Zero123 and other files: - Sets good params for using Zero123! Doesn't use depth or normal images - Changes background for Zero123 to non-trainable solid-color-background with white color - Changes optimizer to Adam - The normals are much smoother. Will affect details like thin fingers, etc. But works well overall, and is very conducive to Phase 2 using tetrahedra-sdf-grid - Changes zero123-prompt-processor to dummy-prompt-processor - Adds resolution_milestones to data/image.py : so the height and width can change with iters. For example in zero123.yaml: data: # threestudio/data/image.py -> SingleImageDataModuleConfig image_path: ./load/images/hamburger_rgba.png height: [128, 256, 512] width: [128, 256, 512] resolution_milestones: [200, 300] - Only loads depth and normal images if lambda_depth or lambda_depth_rel > 0, or lambda_normal > 0. Else doesn't load them, only loads the _rgba image. Also, resizes the images according to resolution_milestones - Adds random_aug to solid-colour-background : uses a random background colour with probability random_aug_prob - Sets bg_color as white, changed ambient_ratio from 0.1 + 0.9*rand to 0.5 + 0.5*rand : didn't notice any difference with this change GEOMETRY REFINEMENT using tetrahedra-sdf-grid - Fixes to dataloaders to be compatible with nvdiffrast - Adds scripts to refine Phase1 results using tetrahedra-sdf-grid : configs/zero123-geometry.yaml and configs/experimental/imagecondition_refine.yaml - Adds normal_consistency loss during refinement - Adds laplacian_smoothness loss during refinement : doesn't work yet, will fix later | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
Don't change image's rgb color - the code (originally from Stable-DreamFusion) that loaded images changed the RGB - `(1 - rgba[..., 3:])` made colors artificially bright (when alpha is between 0 and 1), resulting in halos around the silhouette - pre-multiplying by alpha loses the original color IMHO neither of these transformation are necessary, nor desirable. The colors should be preserved as-is. - also added "bollywood_actress" images, that I generated myself using SD2.2.x. - her silhouette has areas with lots of semi-transparent hair - unfortunately the background segmentation code makes her hair artificially bright as well, but that's a problem for another day - making this changes makes very little difference on Anya and on "Bollywood actress", but I maintain a strong opinion that this is more correct - run_zero123_examples.sh has the commands I ran, although I manually updated the code in between each experiment. | 3 年前 | |
Don't change image's rgb color - the code (originally from Stable-DreamFusion) that loaded images changed the RGB - `(1 - rgba[..., 3:])` made colors artificially bright (when alpha is between 0 and 1), resulting in halos around the silhouette - pre-multiplying by alpha loses the original color IMHO neither of these transformation are necessary, nor desirable. The colors should be preserved as-is. - also added "bollywood_actress" images, that I generated myself using SD2.2.x. - her silhouette has areas with lots of semi-transparent hair - unfortunately the background segmentation code makes her hair artificially bright as well, but that's a problem for another day - making this changes makes very little difference on Anya and on "Bollywood actress", but I maintain a strong opinion that this is more correct - run_zero123_examples.sh has the commands I ran, although I manually updated the code in between each experiment. | 3 年前 | |
Don't change image's rgb color - the code (originally from Stable-DreamFusion) that loaded images changed the RGB - `(1 - rgba[..., 3:])` made colors artificially bright (when alpha is between 0 and 1), resulting in halos around the silhouette - pre-multiplying by alpha loses the original color IMHO neither of these transformation are necessary, nor desirable. The colors should be preserved as-is. - also added "bollywood_actress" images, that I generated myself using SD2.2.x. - her silhouette has areas with lots of semi-transparent hair - unfortunately the background segmentation code makes her hair artificially bright as well, but that's a problem for another day - making this changes makes very little difference on Anya and on "Bollywood actress", but I maintain a strong opinion that this is more correct - run_zero123_examples.sh has the commands I ran, although I manually updated the code in between each experiment. | 3 年前 | |
Don't change image's rgb color - the code (originally from Stable-DreamFusion) that loaded images changed the RGB - `(1 - rgba[..., 3:])` made colors artificially bright (when alpha is between 0 and 1), resulting in halos around the silhouette - pre-multiplying by alpha loses the original color IMHO neither of these transformation are necessary, nor desirable. The colors should be preserved as-is. - also added "bollywood_actress" images, that I generated myself using SD2.2.x. - her silhouette has areas with lots of semi-transparent hair - unfortunately the background segmentation code makes her hair artificially bright as well, but that's a problem for another day - making this changes makes very little difference on Anya and on "Bollywood actress", but I maintain a strong opinion that this is more correct - run_zero123_examples.sh has the commands I ran, although I manually updated the code in between each experiment. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
Zero123: Updates to README (#225) * Minor updates * Adds scripts * Deleted * Update README.md * Update README.md * Minor updates * Adds dog * Replaces Zero123 with Zero123XL * Replaces full paths * Updates README * Minor fix --------- Co-authored-by: Vikram Voleti <vikram@ip-26-0-153-234.us-west-2.compute.internal> | 3 年前 | |
[DRAFT] Zero123 Improvements (#216) Multiple improvements to Zero123 and other files: - Sets good params for using Zero123! Doesn't use depth or normal images - Changes background for Zero123 to non-trainable solid-color-background with white color - Changes optimizer to Adam - The normals are much smoother. Will affect details like thin fingers, etc. But works well overall, and is very conducive to Phase 2 using tetrahedra-sdf-grid - Changes zero123-prompt-processor to dummy-prompt-processor - Adds resolution_milestones to data/image.py : so the height and width can change with iters. For example in zero123.yaml: data: # threestudio/data/image.py -> SingleImageDataModuleConfig image_path: ./load/images/hamburger_rgba.png height: [128, 256, 512] width: [128, 256, 512] resolution_milestones: [200, 300] - Only loads depth and normal images if lambda_depth or lambda_depth_rel > 0, or lambda_normal > 0. Else doesn't load them, only loads the _rgba image. Also, resizes the images according to resolution_milestones - Adds random_aug to solid-colour-background : uses a random background colour with probability random_aug_prob - Sets bg_color as white, changed ambient_ratio from 0.1 + 0.9*rand to 0.5 + 0.5*rand : didn't notice any difference with this change GEOMETRY REFINEMENT using tetrahedra-sdf-grid - Fixes to dataloaders to be compatible with nvdiffrast - Adds scripts to refine Phase1 results using tetrahedra-sdf-grid : configs/zero123-geometry.yaml and configs/experimental/imagecondition_refine.yaml - Adds normal_consistency loss during refinement - Adds laplacian_smoothness loss during refinement : doesn't work yet, will fix later | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
[DRAFT] Zero123 Improvements (#216) Multiple improvements to Zero123 and other files: - Sets good params for using Zero123! Doesn't use depth or normal images - Changes background for Zero123 to non-trainable solid-color-background with white color - Changes optimizer to Adam - The normals are much smoother. Will affect details like thin fingers, etc. But works well overall, and is very conducive to Phase 2 using tetrahedra-sdf-grid - Changes zero123-prompt-processor to dummy-prompt-processor - Adds resolution_milestones to data/image.py : so the height and width can change with iters. For example in zero123.yaml: data: # threestudio/data/image.py -> SingleImageDataModuleConfig image_path: ./load/images/hamburger_rgba.png height: [128, 256, 512] width: [128, 256, 512] resolution_milestones: [200, 300] - Only loads depth and normal images if lambda_depth or lambda_depth_rel > 0, or lambda_normal > 0. Else doesn't load them, only loads the _rgba image. Also, resizes the images according to resolution_milestones - Adds random_aug to solid-colour-background : uses a random background colour with probability random_aug_prob - Sets bg_color as white, changed ambient_ratio from 0.1 + 0.9*rand to 0.5 + 0.5*rand : didn't notice any difference with this change GEOMETRY REFINEMENT using tetrahedra-sdf-grid - Fixes to dataloaders to be compatible with nvdiffrast - Adds scripts to refine Phase1 results using tetrahedra-sdf-grid : configs/zero123-geometry.yaml and configs/experimental/imagecondition_refine.yaml - Adds normal_consistency loss during refinement - Adds laplacian_smoothness loss during refinement : doesn't work yet, will fix later | 3 年前 | |
a lot of minor fixes | 3 年前 | |
a lot of minor fixes | 3 年前 | |
a lot of minor fixes | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
[DRAFT] Zero123 Improvements (#216) Multiple improvements to Zero123 and other files: - Sets good params for using Zero123! Doesn't use depth or normal images - Changes background for Zero123 to non-trainable solid-color-background with white color - Changes optimizer to Adam - The normals are much smoother. Will affect details like thin fingers, etc. But works well overall, and is very conducive to Phase 2 using tetrahedra-sdf-grid - Changes zero123-prompt-processor to dummy-prompt-processor - Adds resolution_milestones to data/image.py : so the height and width can change with iters. For example in zero123.yaml: data: # threestudio/data/image.py -> SingleImageDataModuleConfig image_path: ./load/images/hamburger_rgba.png height: [128, 256, 512] width: [128, 256, 512] resolution_milestones: [200, 300] - Only loads depth and normal images if lambda_depth or lambda_depth_rel > 0, or lambda_normal > 0. Else doesn't load them, only loads the _rgba image. Also, resizes the images according to resolution_milestones - Adds random_aug to solid-colour-background : uses a random background colour with probability random_aug_prob - Sets bg_color as white, changed ambient_ratio from 0.1 + 0.9*rand to 0.5 + 0.5*rand : didn't notice any difference with this change GEOMETRY REFINEMENT using tetrahedra-sdf-grid - Fixes to dataloaders to be compatible with nvdiffrast - Adds scripts to refine Phase1 results using tetrahedra-sdf-grid : configs/zero123-geometry.yaml and configs/experimental/imagecondition_refine.yaml - Adds normal_consistency loss during refinement - Adds laplacian_smoothness loss during refinement : doesn't work yet, will fix later | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
more zero123 challenges + new zero123 defaults (#166) Add zero123 challenges, originally in stable-dreamfusion. Also improve the zero123 configuration to reconstruct some of the challenges. NOTE: all experiments and results were run with zero123XL, a yet-to-be-released model from the Objaverse and Zero123 primary authors. (Thanks Ruoshi and Matt for sharing them with us!) The current official zero123 model will likely not produce as good results and/or take more steps to converge. - Increase camera distance to reduce risk of zero123 cropped predictions, floaters - Reduced rate of white background from 50% to 20% a. IIRC this sped up convergence and improved zero123 predictions, since zero123 is almost only trained on white background b. I tried 10%, and that produced better results for most scenes, except it caused the baby phoenix to temporarily get lots of big white floaters reduced NeRF render resolution from 128x128 to 64x64. Freed up VRAM that I used to increase the batch size to 12 (to fit in 40GB VRAM) and increase the learning rate (5X), for faster/more reliable convergence - Use hashgrid (not progressive). Converges more reliably. - Use accumulate (not alternate) by default. Haven't compared them lately, but accumulate converged more reliably in previous experiments. And the loss was more regular and easier to compare. - Rebalance all the lambdas for faster, more reliable convergence. Now, the lamba_sds dominates the rest. (example for teddy:) - Decrease max_steps from 10000 to 1999. Honestly, 1000 would be enough for all the examples I looked at. | 3 年前 | |
[DRAFT] Zero123 Improvements (#216) Multiple improvements to Zero123 and other files: - Sets good params for using Zero123! Doesn't use depth or normal images - Changes background for Zero123 to non-trainable solid-color-background with white color - Changes optimizer to Adam - The normals are much smoother. Will affect details like thin fingers, etc. But works well overall, and is very conducive to Phase 2 using tetrahedra-sdf-grid - Changes zero123-prompt-processor to dummy-prompt-processor - Adds resolution_milestones to data/image.py : so the height and width can change with iters. For example in zero123.yaml: data: # threestudio/data/image.py -> SingleImageDataModuleConfig image_path: ./load/images/hamburger_rgba.png height: [128, 256, 512] width: [128, 256, 512] resolution_milestones: [200, 300] - Only loads depth and normal images if lambda_depth or lambda_depth_rel > 0, or lambda_normal > 0. Else doesn't load them, only loads the _rgba image. Also, resizes the images according to resolution_milestones - Adds random_aug to solid-colour-background : uses a random background colour with probability random_aug_prob - Sets bg_color as white, changed ambient_ratio from 0.1 + 0.9*rand to 0.5 + 0.5*rand : didn't notice any difference with this change GEOMETRY REFINEMENT using tetrahedra-sdf-grid - Fixes to dataloaders to be compatible with nvdiffrast - Adds scripts to refine Phase1 results using tetrahedra-sdf-grid : configs/zero123-geometry.yaml and configs/experimental/imagecondition_refine.yaml - Adds normal_consistency loss during refinement - Adds laplacian_smoothness loss during refinement : doesn't work yet, will fix later | 3 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 |
