sstable: consolidate reader/writer options

[RaduBerinde]

sstable: remove rawTombstonesOpt

This option does nothing - the flag it sets is not used anymore.

sstable: move FilterMetricsTracker to ReaderOptions

There is no good reason for this option to be a separate
ReaderOption.

sstable: add Writer method to set snapshot pinned stats

We have a private hook that can be used to get a direct reference to
the properties in the writer. We only use this to set the snapshot
pinned stats. We replace this awkward mechanism with just a simple
method to set these stats. There is little reason to go through hoops
to disallow usage of such a method.

sstable: consolidate reader/writer options

There are currently three mechanisms to configure sstable.Reader or
Writer:

• most settings use ReaderOptions/WriterOptions
• some settings use ReaderOption/WriterOption; some of these are
  kept private using a hook through the private package
• some settings call a function that modifies a Writer, also through
  the private package

This commit moves all settings to ReaderOption/WriterOption
structs. Settings which are not allowed to be used outside Pebble are
defined in sstableinternal, which prevents outside callers from
constructing values for them.

sstable: don't store ReaderOptions in Reader

We keep Readers in the table cache so we don't want their size to be
unnecessarily large. We reduce the size by not storing the full
ReaderOptions inside the Reader; many of the fields are only used
inside NewReader.

[cockroach-teamcity]

This change is 

[jbowens]

Reviewed 4 of 4 files at r1, 6 of 6 files at r2, 3 of 3 files at r3, 22 of 33 files at r4.
Reviewable status: 22 of 35 files reviewed, 2 unresolved discussions (waiting on @itsbilal and @RaduBerinde)

---

internal/sstableinternal/options.go line 30 at r4 (raw file):

	// Reader.NewRangeDelIter() should not be fragmented. Used by debug tools to
	// get a raw view of the tombstones contained in an sstable.
	RawTombstones bool

this was already deleted in a previous commit, right?

---

internal/sstableinternal/options.go line 39 at r4 (raw file):

// tools which may be used on multiple databases configured with different
// comparers.
type Comparers map[string]*base.Comparer

should Comparers and Mergers actually be private? It seems like a reasonable thing a sstable reader may want to configure

[RaduBerinde]

Reviewable status: 22 of 35 files reviewed, 2 unresolved discussions (waiting on @itsbilal and @jbowens)

---

internal/sstableinternal/options.go line 39 at r4 (raw file):

Previously, jbowens (Jackson Owens) wrote…

should Comparers and Mergers actually be private? It seems like a reasonable thing a sstable reader may want to configure

I guess not.. I did that because they were not in options already. I will move them.

By the way, why do we care about the merger at all? We don't seem to be using it anywhere

[RaduBerinde]

TFTR!

Reviewable status: 0 of 35 files reviewed, 2 unresolved discussions (waiting on @itsbilal and @jbowens)

---

internal/sstableinternal/options.go line 30 at r4 (raw file):

Previously, jbowens (Jackson Owens) wrote…

this was already deleted in a previous commit, right?

Oops, fixed.

---

internal/sstableinternal/options.go line 39 at r4 (raw file):

Previously, RaduBerinde wrote…

I guess not.. I did that because they were not in options already. I will move them.

By the way, why do we care about the merger at all? We don't seem to be using it anywhere

Done.