Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

WIP: Reimplementing search_dates#945

Open
gavishpoddar wants to merge 44 commits intoscrapinghub:master from
gavishpoddar:search_dates
Open

WIP: Reimplementing search_dates #945
gavishpoddar wants to merge 44 commits intoscrapinghub:master from
gavishpoddar:search_dates

Conversation

@gavishpoddar
Copy link
Contributor

@gavishpoddar gavishpoddar commented Jul 16, 2021 •
edited
Loading

Reimplementing and simplifying search_dates

A reimplemented and simplified search_dates which more directly uses dateparser.parse, improves accuracy and fixes many bugs

New Feature:

  • search_first_date - searches and returns the first date from the given text.

NOTE: This PR is inspired by the previous implementation of search_dates and #931.

TODO

  • Gather feedback on the API and it's functions.
  • Write docs.
  • Write tests.
  • Chack search translation joints.
  • Fix DATE_ORDER

Copy link
Contributor Author

gavishpoddar commented Jul 16, 2021 •
edited
Loading

This PR will also solve many issues with the search_dates.

Somewhat address.

Some other issue fixed:

2021年08月04日T14:21:37+05:30

would parse 05:30 only after this PR

2021年08月04日T14:21:37 and 05:30

NOTE: THIS LIST IS NO LONGER ACCURATE ANYMORE

Copy link
Contributor Author

Hi, I need a suggestion should I use translated chunks or the original chunks to further parse the data objects. I have currently used translated chunks instead of original chunks as this increased accuracy in some basic tests.

Thanks, and please suggest.

@gavishpoddar gavishpoddar changed the title (削除) [WIP] Reimplementing search_dates & fixing search translation (削除ここまで) (追記) [WIP] Reimplementing search_dates & fixing search translation (追記ここまで) Jul 17, 2021
Copy link
Collaborator

noviluni commented Jul 19, 2021 •
edited
Loading

Hi, I need a suggestion should I use translated chunks or the original chunks to further parse the data objects. I have currently used translated chunks instead of original chunks as this increased accuracy in some basic tests.

Thanks, and please suggest.

Hi @gavishpoddar, using translated chunks makes sense. 👍

Copy link

codecov bot commented Jul 19, 2021 •
edited
Loading

Codecov Report

Base: 98.23% // Head: 98.10% // Decreases project coverage by -0.12% ⚠️

Coverage data is based on head (99e66c6) compared to base (0f5d9c5).
Patch coverage: 96.12% of modified lines in pull request are covered.

❗ Current head 99e66c6 differs from pull request most recent head 2935aae. Consider uploading reports for the commit 2935aae to get more accurate results

Additional details and impacted files 
@@ Coverage Diff @@
## master #945 +/- ##
==========================================
- Coverage 98.23% 98.10% -0.13% 
==========================================
 Files 232 235 +3 
 Lines 2604 2692 +88 
==========================================
+ Hits 2558 2641 +83 
- Misses 46 51 +5
Impacted Files Coverage Δ
dateparser/search/search.py 94.73% <94.33%> (-4.64%) ⬇️
dateparser/search/__init__.py 100.00% <100.00%> (ø)
dateparser/search/languages.py 100.00% <100.00%> (ø)
dateparser/date_parser.py 93.75% <0.00%> (-0.19%) ⬇️
dateparser/date.py 99.24% <0.00%> (-0.04%) ⬇️
dateparser/parser.py 99.01% <0.00%> (-0.01%) ⬇️
dateparser/utils/__init__.py 95.27% <0.00%> (ø)
dateparser/languages/locale.py 98.69% <0.00%> (ø)
dateparser/conf.py 100.00% <0.00%> (ø)
dateparser/__init__.py 100.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Contributor Author

Hi @noviluni, I have made the changes for tests to work. Can you please approve the workflow approval?

Copy link
Member

lopuhin commented Jul 22, 2021

@gavishpoddar workflows approved 👍
gavishpoddar reacted with thumbs up emoji

Copy link
Contributor Author

Hi @noviluni, @lopuhin, @kishan3
The new search_dates implementation is nearing its completion.

Code reviews would be great to receive.

Copy link
Contributor Author

I need help with one test

StaticTzInfo 'UTC' is expected.
UTC is received from new search_dates

Copy link
Contributor Author

gavishpoddar commented Aug 6, 2021 •
edited
Loading

Hi after some changes the codes are very much compatible with the old search_dates. Some tests are modified as the working of the search_dates has been changed.

Currently, this PR is left with some docs changes (just replacing the current with new docs and search_first_date).

@gavishpoddar gavishpoddar changed the title (削除) [WIP] Reimplementing search_dates & fixing search translation (削除ここまで) (追記) Reimplementing search_dates & fixing search translation (追記ここまで) Aug 6, 2021
Copy link
Contributor Author

Replaced previous search_dates with the new implimentaion

Copy link
Member

@lopuhin lopuhin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gavishpoddar , I left some comments regarding tests. Please tell if you need advice with how to implement xfail.

Copy link
Member

@lopuhin lopuhin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gavishpoddar a few docs suggestions

Copy link
Contributor Author

Hey, this PR is updated with the #932

lopuhin reacted with thumbs up emoji

Copy link

Eckii24 commented Aug 6, 2022

Hi @gavishpoddar,

I found your PR to improve the behavior of search_dates.
There has been no further progress for almost a year. What is the current status of this PR?

Is there anything I can do to help get this PR merged?
davidjb99 and jan-kubica reacted with thumbs up emoji

def __init__(self):
self.loader = LocaleDataLoader()
self.available_language_map = self.loader.get_locale_map()
self.search = _ExactLanguageSearch(self.loader)
Copy link
Member

@Gallaecio Gallaecio Dec 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The removal of DateSearchWithDetection.search is backward-incompatible.

Copy link
Contributor Author

@gavishpoddar gavishpoddar Dec 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can create a shortcut the make DateSearchWithDetection.search and add a deprecation warning or simply rename.

Please suggest a preferred action.

Copy link
Member

@Gallaecio Gallaecio Dec 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think keeping the old objects around with their old names, logging a warning when used (i.e. by exposing them through a property whose getter logs a warning), would be ideal.

@gavishpoddar gavishpoddar changed the title (削除) Reimplementing search_dates (削除ここまで) (追記) WIP: Reimplementing search_dates (追記ここまで) Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lopuhin lopuhin lopuhin left review comments

@Gallaecio Gallaecio Gallaecio left review comments

@noviluni noviluni noviluni requested changes

@wRAR wRAR Awaiting requested review from wRAR

@serhii73 serhii73 Awaiting requested review from serhii73

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /