Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: document truncation and loss in notion document sync #5631

Merged
merged 1 commit into from
Jul 5, 2024

Conversation

Aurelius-Huang
Copy link
Contributor

@Aurelius-Huang Aurelius-Huang commented Jun 26, 2024

Description

notion extractor only retrieves the first page of many blocks, and the subsequent blocks are lost.

According to the introduction of Pagination in the Notion Developers document, when the number of Blocks contained in a Pagination exceeds 100, it is necessary to obtain them in a paginated manner to get the complete content of the Notion Page.

However, the acquisition method in notion_extractor.py can only successfully obtain the first blocks page of the Notion Page (up to 100). It is not difficult to find out from the Notion Developers document that the reason is that when calling https://api.notion.com/v1/blocks/{block_id}/children, the start_cursor of the next page is mistakenly passed as block_id, while in fact start_cursor is passed through the Query Params of the GET request.

image

In addition, the parameter transmission method of the Query Params of the GET request is also wrong (formal parameter: json -> params).

image

Fixes # (issue)

Type of Change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Find a longer Notion Page (with more than 100 Blocks) and perform the Sync from Notion operation in Knowledge to verify that the version after this PR can synchronize the complete Notion Page content, while the previous version can only obtain the content of the first 100 Blocks, and the other content is lost.

  • TODO

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
  • optional I have made corresponding changes to the documentation
  • optional I have added tests that prove my fix is effective or that my feature works
  • optional New and existing unit tests pass locally with my changes
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. 🐞 bug Something isn't working labels Jun 26, 2024
@Aurelius-Huang
Copy link
Contributor Author

@JohnJyong Could you please review this PR? Is the problem described in this PR not clear enough to reproduce the problem?

@JohnJyong
Copy link
Contributor

LGTM , thanks for you contribution @Aurelius-Huang

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jul 5, 2024
@JohnJyong JohnJyong merged commit f546db5 into langgenius:main Jul 5, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐞 bug Something isn't working lgtm This PR has been approved by a maintainer size:M This PR changes 30-99 lines, ignoring generated files.
2 participants