Where Self Meets Sized: Revisiting Object Safety

The concept of object safety in Rust was recently refined to be more flexible in an important way: the checks can be disabled for specific methods by using where clauses to restrict them to only work when Self: Sized.

This post is a rather belated fourth entry in my series on trait objects and object safety: Peeking inside Trait Objects, The Sized Trait and Object Safety. It’s been long enough that a refresher is definitely in order, although this isn’t complete coverage of the details.

Recap

Rust offers open sets of types, type erasure and dynamic dispatch via trait objects. However, to ensure a uniform handling of trait objects and non-trait objects in generic code, there are certain restrictions about exactly which traits can be used to create objects: this is object safety.

A trait is object safe only if the compiler can automatically implement it for itself, by implementing each method as a dynamic function call through the vtable stored in a trait object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
trait Foo {
    fn method_a(&self) -> u8;

    fn method_b(&self, x: f32) -> String;
}

// automatically inserted by the compiler
impl<'a> Foo for Foo+'a {
    fn method_a(&self) -> u8 {
         // dynamic dispatch to `method_a` of erased type
         self.method_a()
    }
    fn method_b(&self, x: f32) -> String {
         // as above
         self.method_b(x)
    }
}

Without the object safety rules one can write functions with type signatures satisfied by trait objects, where the internals make it impossible to actually use with trait objects. However, Rust tries to ensure that this can’t happen—code should only need to know the signatures of anything it calls, not the internals—and hence object safety.

These rules outlaw creating trait objects of, for example, traits with generic methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
trait Bar {
    fn bad<T>(&self, x: T);
}

impl Bar for u8 {
    fn bad<T>(&self, _: T) {}
}

fn main() {
    &1_u8 as &Bar;
}

/*
...:10:5: 10:7 error: cannot convert to a trait object because trait `Bar` is not object-safe [E0038]
...:10     &1 as &Bar;
           ^~
...:10:5: 10:7 note: method `bad` has generic type parameters
...:10     &1 as &Bar;
           ^~
*/

Trait object values always appear behind a pointer, like &SomeTrait or Box<AnotherTrait>, since the trait value “SomeTrait” itself doesn’t have size known at compile time. This property is captured via the Sized trait, which is implemented for types like i32, or simple structs and enums, but not for unsized slices [T], or the plain trait types SomeTrait.

Iterating on the design

One impact of introducing object safety was that the design of several traits had to change. The most noticeable ones were Iterator, and the IO traits Read and Write (although they were probably Reader and Writer at that point).

Focusing on the former, before object safety it was defined something⁰ like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
trait Iterator {
    type Item;

    fn next(&mut self) -> Option<Self::Item>;
    fn size_hint(&self) -> (usize, Option<usize>) { /* ... */ }

    // ... methods methods methods ...

    fn zip<U>(self, other: U) -> Zip<Self, U::IntoIter>
        where U: IntoIterator
    { /* ... */ }

    fn map<B, F>(self, f: F) -> Map<Self, F>
        where F: FnMut(Self::Item) -> B
    { /* ... */ }

    // etc
}

The above Iterator isn’t object safe: it has generic methods, and so it isn’t possible to implement Iterator for Iterator itself. This is unfortunate, since it is very useful to be able to create and use Iterator trait objects, so it had to be made object safe.

The solution at the time was extension traits: define a new trait IteratorExt that incorporated all the object unsafe methods, and use a blanket implementation to implement it for all Iterators “from the outside”.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
trait Iterator {
    type Item;

    fn next(&mut self) -> Option<Self::Item>;

    fn size_hint(&self) -> (usize, Option<usize>) { /* ... */ }
}

trait IteratorExt: Sized + Iterator {
    // ... methods methods methods ...

    fn zip<U>(self, other: U) -> Zip<Self, U::IntoIter>
        where U: IntoIterator
    { /* ... */ }

    fn map<B, F>(self, f: F) -> Map<Self, F>
        where F: FnMut(Self::Item) -> B
    { /* ... */ }

    // etc
}
// blanket impl, for all iterators
impl<I: Iterator> IteratorExt for I {}

The next and size_hint methods are object safe, so this version of Iterator can create trait objects: Box<Iterator<Item = u8>> is a legal iterator over bytes. It works because the methods of IteratorExt are no longer part of Iterator and so they’re not involved in any object considerations for it.

Fortunately, those methods aren’t lost on trait objects, because there are implementations like the following, allowing the blanket implementation of IteratorExt to kick in:

1
2
3
4
5
6
7
8
9
10
11
12
// make Box<...> an Iterator by deferring to the contents
impl<I: Iterator + ?Sized> Iterator for Box<I> {
    type Item = I::Item;

    fn next(&mut self) -> Option<I::Item> {
        (**self).next()
    }

    fn size_hint(&self) -> (usize, Option<usize>) {
        (**self).size_hint()
    }
}

(The ?Sized ensures this applies to Box<Iterator<...>> trait objects as well as simply Box<SomeType> where SomeType is a normal type that implements Iterator.)

This approach has some benefits, like clarifying the separation between the “core” methods (next and size_hint) and the helpers. However, it has several downsides, especially for cases that aren’t Iterator:

• extra traits in the documentation,
• users will have to import those extra traits
• it only works with default-able methods,
• the defaults can’t be overridden, e.g. there’s no way for a specific type to slot in a more efficient way to implement a method

All-in-all, it was a wet blanket on libraries. Fortunately, not all was lost: let’s meet our saviour.

It’s a bird… it’s a plane…

It’s a where clause!

where clauses allow predicating functions/methods/types on essentially arbitrary trait relationships, not just the plain <T: SomeTrait>, where the left-hand side has to be a generic type declared right then and there. For example, one can use From to convert to types with a where clause.

1
2
3
4
5
fn convert_to_string<T>(x: T) -> String
    where String: From<T>
{
    String::from(x)
}

The important realisation was that where allows placing restrictions on Self directly on methods, so that certain methods only exist for some implementing types. This was used to great effect to collapse piles of traits into a single one, for example in std::iter. Rust 0.12.0 had a swathe of extra Iterator traits: Additive..., Cloneable..., Multiplicative..., MutableDoubleEnded..., Ord....

Each of these were designed to define a few extra methods that required specific restrictions on the element type of the iterator, for example, OrdIterator needed Ord elements:

1
2
3
4
5
6
7
8
9
10
trait OrdIterator: Iterator {
     fn max(&mut self) -> Option<Self::Item>;
     // ...
}

impl<A: Ord, I: Iterator<Item = A>> OrdIterator for I {
    fn max(&mut self) -> Option<A> { /* ... */ }

    // ...
}

The current std::iter is much cleaner: all the traits above have been merged into Iterator itself with where clauses, e.g. max:

1
2
3
4
5
6
7
8
9
10
11
trait Iterator {
    type Item;

    // ...

    fn max(self) -> Option<Self::Item>
        where Self::Item: Ord
    { /* ... */ }

    // ...
}

Notably, there’s no restriction on Item for general Iterators, only on max, so iterators retain full flexibility while still gaining a max method that only works when it should:

1
2
3
4
5
6
7
8
9
10
11
struct NotOrd;

fn main() {
    (0..10).max(); // ok
    (0..10).map(|_| NotOrd).max();
}
/*
...:5:29: 5:34 error: the trait `core::cmp::Ord` is not implemented for the type `NotOrd` [E0277]
...:5     (0..10).map(|_| NotOrd).max();
                                  ^~~~~
*/

This approach works fine for normal traits like Ord, and also works equally well for “special” traits like Sized: it is possible to restrict methods to only work when Self has a statically known size with where Self: Sized. Initially this had no interaction with object safety, it would just influence what exactly that method could do.

Putting it together

The piece that interacts with object safety is RFC 817, which made where Self: Sized special: the compiler now understands that methods tagged with that cannot ever be used on a trait object, even in generic code. This means it is perfectly correct to completely ignores any methods with that where clause when checking object safety.

The bad example from the start can be written to compile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
trait Bar {
    fn bad<T>(&self, x: T)
        where Self: Sized;
}

impl Bar for u8 {
    fn bad<T>(&self, _: T)
        where Self: Sized
    {}
}

fn main() {
    &1_u8 as &Bar;
}

And also adjusted to not compile: try calling (&1_u8 as &Bar).bad("foo") in main and the compiler spits out an error,

1
2
3
4
5
6
...:13:21: 13:31 error: the trait `core::marker::Sized` is not implemented for the type `Bar` [E0277]
...:13     (&1_u8 as &Bar).bad("foo")
                           ^~~~~~~~~~
...:13:21: 13:31 note: `Bar` does not have a constant size known at compile-time
...:13     (&1_u8 as &Bar).bad("foo")
                           ^~~~~~~~~~

Importantly, this solves the Iterator problem: there’s no longer a need to split methods into extension traits to ensure object safety, one can instead just guard the bad ones. Iterator now looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
trait Iterator {
    type Item;

    fn next(&mut self) -> Option<Self::Item>;

    fn size_hint(&self) -> (usize, Option<usize>) { /* ... */ }

    // ... methods methods methods ...

    fn zip<U>(self, other: U) -> Zip<Self, U::IntoIter>
        where Self: Sized, U: IntoIterator
    { /* ... */ }

    fn map<B, F>(self, f: F) -> Map<Self, F>
        where Self: Sized, F: FnMut(Self::Item) -> B
    { /* ... */ }

    // etc
}

(Along with max and the other where-reliant methods from the other *Iterator traits mentioned above.)

The extra flexibility this where clauses offer is immensely helpful for designing that perfect API. Of course, just adding where Self: Sized isn’t a complete solution or the only trick: the current Iterator still has the same sort of implementations of Iterator for Box<I> where I: Iterator + ?Sized, and traits using the where technique may want to adopt others that Iterator does.