Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datamon 2.0 WAL #271

Merged
merged 1 commit into from
Oct 28, 2019
Merged

Datamon 2.0 WAL #271

merged 1 commit into from
Oct 28, 2019

Conversation

kerneltime
Copy link
Contributor

@kerneltime kerneltime commented Oct 17, 2019 •
edited
Loading

This is a WIP commit for the WAL.
ToDo:

  • Add test for parallel listing of WAL

  • Add documentation

Signed-off-by: Ritesh H Shukla [email protected]

@kerneltime
Copy link
Contributor Author

The build failure is due to go mod updates, will fix it.

ransomw1c
ransomw1c reviewed Oct 17, 2019
View reviewed changes
pkg/wal/wal.go Outdated Show resolved Hide resolved
ransomw1c
ransomw1c reviewed Oct 17, 2019
View reviewed changes
pkg/wal/wal.go Outdated Show resolved Hide resolved
@ransomw1c
Copy link
Contributor

the parallelism setup based on count

count

is considered super

fredbi
fredbi reviewed Oct 19, 2019
View reviewed changes
go.mod Outdated Show resolved Hide resolved
go.mod Outdated Show resolved Hide resolved
pkg/core/bundle_list_test.go Outdated Show resolved Hide resolved
pkg/storage/gcs/store_test.go Outdated Show resolved Hide resolved
pkg/storage/gcs/store_test.go Outdated Show resolved Hide resolved
pkg/storage/store.go Outdated Show resolved Hide resolved
pkg/wal/model.go
}
}

func Unmarshal(b []byte) (*Entry, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should rather implement here the YAML unmarshaller interface: https://github.com/go-yaml/yaml/blob/v2.2.4/yaml.go#L32

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Will do it.

pkg/wal/wal.go Show resolved Hide resolved
pkg/wal/wal.go Outdated Show resolved Hide resolved
@kerneltime kerneltime changed the title [WIP] Datamon 2.0 WAL Datamon 2.0 WAL Oct 23, 2019
@kerneltime kerneltime force-pushed the wal-wip branch 12 times, most recently from ed29914 to 3566870 Compare October 25, 2019 05:54
@ransomw1c
Copy link
Contributor

approval pre-supposes Fred's comments re. yaml and type alias are to be addressed..

.. the .md port of the Confluence is a welcome addition.

my partial paraphrase is

to ensure bundle descriptor upload and WAL entries occur according to the same timer, we rely on urls with expiration and GCS being attached to the same timer

here, the wal is landing as a coherent idea before attaching the wal to bundle descriptor uploads

Datamon 2.0 WAL
f683fc7 
This is a WIP commit for the WAL.

Signed-off-by: Ritesh H Shukla <[email protected]>
@kerneltime kerneltime mentioned this pull request Oct 28, 2019
Open
@kerneltime
Copy link
Contributor Author

Will take the YAML interfaces on later #277

@kerneltime kerneltime merged commit 4d09e60 into master Oct 28, 2019
@kerneltime kerneltime deleted the wal-wip branch November 14, 2019 22:41
@ransomw1c ransomw1c mentioned this pull request Mar 12, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fredbi fredbi fredbi approved these changes

@ransomw1c ransomw1c ransomw1c approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@kerneltime @ransomw1c @fredbi