Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ThinLTO][NFC] Refactor FileCache #110463

Merged
merged 2 commits into from
Oct 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion llvm/include/llvm/LTO/LTO.h
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@ class LTO {
///
/// The client will receive at most one callback (via either AddStream or
/// Cache) for each task identifier.
Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
Error run(AddStreamFn AddStream, FileCache Cache = {});

/// Static method that returns a list of libcall symbols that can be generated
/// by LTO but might not be visible from bitcode symbol table.
Expand Down
47 changes: 37 additions & 10 deletions llvm/include/llvm/Support/Caching.h
Original file line number Diff line number Diff line change
Expand Up @@ -41,21 +41,48 @@ class CachedFileStream {
using AddStreamFn = std::function<Expected<std::unique_ptr<CachedFileStream>>(
unsigned Task, const Twine &ModuleName)>;

/// This is the type of a file cache. To request an item from the cache, pass a
/// unique string as the Key. For hits, the cached file will be added to the
/// link and this function will return AddStreamFn(). For misses, the cache will
/// return a stream callback which must be called at most once to produce
/// content for the stream. The file stream produced by the stream callback will
/// add the file to the link after the stream is written to. ModuleName is the
/// unique module identifier for the bitcode module the cache is being checked
/// for.
/// This is a callable that manages file caching operations. It accepts a task
/// ID \p Task, a unique key \p Key, and a module name \p ModuleName, and
/// returns AddStreamFn(). This function determines whether a cache hit or miss
/// occurs and handles the appropriate actions.
using FileCacheFunction = std::function<Expected<AddStreamFn>(
unsigned Task, StringRef Key, const Twine &ModuleName)>;

/// This type represents a file cache system that manages caching of files.
/// It encapsulates a caching function and the directory path where the cache is
/// stored. To request an item from the cache, pass a unique string as the Key.
/// For hits, the cached file will be added to the link and this function will
/// return AddStreamFn(). For misses, the cache will return a stream callback
/// which must be called at most once to produce content for the stream. The
/// file stream produced by the stream callback will add the file to the link
/// after the stream is written to. ModuleName is the unique module identifier
/// for the bitcode module the cache is being checked for.
///
/// Clients generally look like this:
///
/// if (AddStreamFn AddStream = Cache(Task, Key, ModuleName))
/// ProduceContent(AddStream);
using FileCache = std::function<Expected<AddStreamFn>(
unsigned Task, StringRef Key, const Twine &ModuleName)>;
///
/// CacheDirectoryPath stores the directory path where cached files are kept.
struct FileCache {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move the docs for FileCacheFunction to FileCache since I think that will be the primary struct we should use. Also, we should add brief docs for FileCacheFunction to explain the arguments.

FileCache(FileCacheFunction CacheFn, const std::string &DirectoryPath)
: CacheFunction(std::move(CacheFn)), CacheDirectoryPath(DirectoryPath) {}
FileCache() = default;

Expected<AddStreamFn> operator()(unsigned Task, StringRef Key,
const Twine &ModuleName) {
assert(isValid() && "Invalid cache function");
return CacheFunction(Task, Key, ModuleName);
}
const std::string &getCacheDirectoryPath() const {
return CacheDirectoryPath;
}
bool isValid() const { return static_cast<bool>(CacheFunction); }

private:
FileCacheFunction CacheFunction = nullptr;
std::string CacheDirectoryPath;
};

/// This type defines the callback to add a pre-existing file (e.g. in a cache).
///
Expand Down
2 changes: 1 addition & 1 deletion llvm/lib/LTO/LTO.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1483,7 +1483,7 @@ class InProcessThinBackend : public ThinBackendProc {
return E;
}

if (!Cache || !CombinedIndex.modulePaths().count(ModuleID) ||
if (!Cache.isValid() || !CombinedIndex.modulePaths().count(ModuleID) ||
all_of(CombinedIndex.getModuleHash(ModuleID),
[](uint32_t V) { return V == 0; }))
// Cache disabled or no entry for this module in the combined index or
Expand Down
5 changes: 3 additions & 2 deletions llvm/lib/Support/Caching.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ Expected<FileCache> llvm::localCache(const Twine &CacheNameRef,
TempFilePrefixRef.toVector(TempFilePrefix);
CacheDirectoryPathRef.toVector(CacheDirectoryPath);

return [=](unsigned Task, StringRef Key,
const Twine &ModuleName) -> Expected<AddStreamFn> {
auto Func = [=](unsigned Task, StringRef Key,
const Twine &ModuleName) -> Expected<AddStreamFn> {
// This choice of file name allows the cache to be pruned (see pruneCache()
// in include/llvm/Support/CachePruning.h).
SmallString<64> EntryPath;
Expand Down Expand Up @@ -167,4 +167,5 @@ Expected<FileCache> llvm::localCache(const Twine &CacheNameRef,
Task);
};
};
return FileCache(Func, CacheDirectoryPathRef.str());
}
Loading