A log collector for Android

Cookpad

Last update: Dec 19, 2022

Related tags

Utility puree-android

Overview

Puree

Description

Puree is a log collector which provides the following features:

Filtering: Enable to interrupt process before sending log. You can add common params to logs, or the sampling of logs.
Buffering: Store logs to buffers and send them later.
Batching: Send logs in a single request with PureeBufferedOutput.
Retrying: Retry to send logs after backoff time if sending logs fails.

Puree helps you unify your logging infrastructure.

Installation

This is published on jitpack and you can use Puree as:

// build.gradle
buildscript {
    repositories {
        maven { url 'https://jitpack.io' }
    }
    ...
}

// app/build.gradle
dependencies {
    implementation "com.github.cookpad:puree-android:$latestVersion"
}

Usage

Initialize

Configure Puree with PureeConfiguration in Application#onCreate(), which registers pairs of what and where.

public class MyApplication extends Application {
    @Override
    public void onCreate() {
        Puree.initialize(buildConfiguration(this));
    }

    public static PureeConfiguration buildConfiguration(Context context) {
        PureeFilter addEventTimeFilter = new AddEventTimeFilter();
        return new PureeConfiguration.Builder(context)
                .pureeSerializer(new PureeGsonSerializer())
                .executor(Executors.newScheduledThreadPool(1)) // optional
                .register(ClickLog.class, new OutLogcat())
                .register(ClickLog.class, new OutBufferedLogcat().withFilters(addEventTimeFilter))
                .build();
    }
}

Definition of PureeLog objects

Puree requires that clients supply an implementation of PureeSerializer to be able to serialize the logs. For instance, this is an implementation that uses Gson parser:

public class PureeGsonSerializer implements PureeSerializer {
    private Gson gson = new Gson();

    @Override
    public String serialize(Object object) {
        return gson.toJson(object);
    }
}

A log class is just a POJO whose properties are annotated following the requirements of the Json parser that you provided with PureeSerializer.

public class ClickLog {
    @SerializedName("page")
    private String page;
    @SerializedName("label")
    private String label;

    public ClickLog(String page, String label) {
        this.page = page;
        this.label = label;
    }
}

You can use Puree.send() to send these logs to registered output plugins:

Puree.send(new ClickLog("MainActivity", "Hello"));
// => {"page":"MainActivity","label":"Hello"}

Definition of PureeOutput plugins

There are two types of output plugins: non-buffered and buffered.

PureeOutput: Non-buffered output plugins write logs immediately.
PureeBufferedOutput: Buffered output plugins enqueue logs to a local storage and then flush them in background tasks.

If you don't need buffering, you can use PureeOutput.

public class OutLogcat extends PureeOutput {
    private static final String TYPE = "out_logcat";

    @Override
    public String type() {
        return TYPE;
    }

    @Override
    public OutputConfiguration configure(OutputConfiguration conf) {
        return conf;
    }

    @Override
    public void emit(JsonObject jsonLog) {
        Log.d(TYPE, jsonLog.toString());
    }
}

If you need buffering, you can use PureeBufferedOutput.

public class OutFakeApi extends PureeBufferedOutput {
    private static final String TYPE = "out_fake_api";

    private static final FakeApiClient CLIENT = new FakeApiClient();

    @Override
    public String type() {
        return TYPE;
    }

    @Override
    public OutputConfiguration configure(OutputConfiguration conf) {
        // you can change settings of this plugin
        // set interval of sending logs. defaults to 2 * 60 * 1000 (2 minutes).
        conf.setFlushIntervalMillis(1000);
        // set num of logs per request. defaults to 100.
        conf.setLogsPerRequest(10);
        // set retry count. if fail to send logs, logs will be sending at next time. defaults to 5.
        conf.setMaxRetryCount(3);
        return conf;
    }

    @Override
    public void emit(JsonArray jsonArray, final AsyncResult result) {
        // you have to call result.success or result.fail()
        // to notify whether if puree can clear logs from buffer
        CLIENT.sendLog(jsonArray, new FakeApiClient.Callback() {
            @Override
            public void success() {
                result.success();
            }

            @Override
            public void fail() {
                result.fail();
            }
        });
    }
}

Definition of Filters

If you need to add common params to each logs, you can use PureeFilter:

public class AddEventTimeFilter implements PureeFilter {
    public JsonObject apply(JsonObject jsonLog) {
        jsonLog.addProperty("event_time", System.currentTimeMillis());
        return jsonLog;
    }
}

You can make PureeFilter#apply() to return null to skip sending logs:

public class SamplingFilter implements PureeFilter {
    private final float samplingRate;

    public SamplingFilter(float samplingRate) {
        this.samplingRate = samplingRate;
    }

    @Override
    public JsonObject apply(JsonObject jsonLog) {
        return (samplingRate < Math.random() ? null : jsonLog);
    }
}

Then register filters to output plugins on initializing Puree.

new PureeConfiguration.Builder(context)
        .register(ClickLog.class, new OutLogcat())
        .register(ClickLog.class, new OutFakeApi().withFilters(addEventTimeFilter, samplingFilter)
        .build();

Purging old logs

To discard unnecessary recorded logs, purge age can be configured in the PureeBufferedOutput subclass.

public class OutPurge extends PureeBufferedOutput {

    // ..

    @Override
    public OutputConfiguration configure(OutputConfiguration conf) {
        // set to purge buffered logs older than 2 weeks
        conf.setPurgeAgeMillis(2 * 7 * 24 * 60 * 60 * 1000);
        return conf;
    }
}

The configured storage must extend EnhancedPureeStorage to support purging of old logs. The default PureeSQLiteStorage can be used to enable this feature.

new PureeConfiguration.Builder(context)
        .storage(new PureeSQLiteStorage(context))
        .register(ClickLog.class, new OutPurge().withFilters(addEventTimeFilter, samplingFilter)
        .build();

Testing

If you want to mock or ignore Puree.send() and Puree.flush(), you can use Puree.setPureeLogger() to replace the internal logger. See PureeTest.java for details.

Release Engineering

Update CHANGES.md
git tag $latest_version
git push origin $latest_version

Copyright

See LICENSE.txt for the license.

Comments

[v2] New interfaces to configure Puree

I'd like to change interfaces to configure Puree. Because

I don't want to specify TAG when I send log
It's not easy to understand where log is sent

Before

Destination definition is far apart from sender.

new PureeConfiguration.Builder(context)
    .registerOutput(new OutDisplay())
    .registerOutput(new OutBufferedLogcat(), addEventTimeFilter, samplingFilter)
    .registerOutput(new OutLogcat(), addEventTimeFilter)
    .build();

I need to know where I should send logs. In addition, It's not good that I have to specify output plugin by String.

Puree.send(new ClickLog("MainActivity", "BUTTON 1"), OutDisplay.TAG);
Puree.send(new ClickLog("MainActivity", "BUTTON 2"), OutDisplay.TAG);
...

After

It's easy to understand where log is sent at a glance.

new PureeConfiguration.Builder(context)
    .source(ClickLog.class).to(new OutDisplay())
    .source(ClickLog.class).filters(addEventTimeFilter, samplingFilter).to(new OutBufferedLogcat())
    .source(PvLog.class).filter(addEventTimeFilter).to(new OutLogcat())
    .build();

Puree.send(new ClickLog("MainActivity", "BUTTON 1"));
Puree.send(new ClickLog("MainActivity", "BUTTON 2"));
...

It looks simple and intuitive.

screen shot 2015-02-21 at 10 18 41

Todo

[x] Change interfaces
[x] Add tests
[x] Update documents
[x] Update v2.0.0

opened by rejasupotaro 6

Tasks package's classes are better to extend IntentService
LogHouse Looks good to me:house_with_garden:

I think tasks package's classes are better to extend IntentService, because

AsncTask does not correspond to the life cycle of the Fragment and Activity.

There is no need to return the results.

How do you think?
opened by hotchemi 6
Add Puree#truncateBufferedLogs()
First of all, thank you for this great library. It is very useful. I would like to add a new feature to this library. This feature discards old buffered data from a device. Of course you can set a threshold to control an amount of logs you want to discard. As one possible use case, we can avoid storing too much data in users' devices.

Usage

Set the delete threshold at PureeConfigurator.java
Call PureeConfiguration#deleteThreshold() method to set the threshold.
If the threshold is not set , INTEGER_MAX will be set as default. (which means data will never be discarded)

public class PureeConfigurator { private static final int DELETE_THRESHOLD = 1000; public static void configure(Context context) { Puree.initialize(buildConf(context)); } public static PureeConfiguration buildConf(Context context){ PureeFilter addEventTimeFilter = new AddPredefinedEventFilter(context); PureeConfiguration conf = new PureeConfiguration.Builder(context) .register(EventLog.class, new OutBufferedLogcat().withFilters(addEventTimeFilter)) .deleteThreshold(DELETE_THRESHOLD) .build(); conf.printMapping(); return conf; } }

Call Puree#discardOldBufferedLogs() any where you want to discard old logs
Launch Activity might be the best place to call this method. With an example below, any logs that are over DELETE_THRESHOLD will be discarded.

@Override public void onCreate(Bundle savedInstanceState) { ... Puree.discardOldBufferedLogs(); }
opened by tomoima525 5
v3: get rid of org.json and use Gson for performance
https://github.com/cookpad/puree-android/issues/31

[x] get rid of org.json

[x] benchmark with v2 - about 2 times faster in average

I think it is good enough

[x] documentation

benchmark

Note it is not fair enough because Puree.flush() invokes asynchronous tasks. That is because the first try is so fast.

v2

v3
opened by gfx 5
Remove the dependency to Gson

Hello, Thanks for that nice library. But Gson is not the only one library to handle JSON, and as is well known, using several libraries for the same purpose is not fitting to build apks, so implicitly depending to such a library is not good.

I remove the dependency to Gson by default because of above, though the compatibility is lost. I hope the implicit dependency is removed anyway, so this request is simply to acknowledge that.

Thanks.

opened by k24 5
Remove Gson dependency and PureeLog interface

This PR removes Gson dependency from Puree and replaces the types used for representing the serialized JSON, JsonObject and the JsonArray, by String and List<String> respectively. After doing this change, PureeLog was of no use so it has conveniently been removed.

The instrumentation tests, as such as the demo project and the docs, have been properly updated to reflect the latest changes

opened by VictorAlbertos 4
Prepare for Javadoc publishing
What

[x] Add task which publishes Javadoc jar to artifacts. db8005f

[x] Fix errors and warnings occurred in a Javadoc publish task. 712bdb5

[x] Add shell and gradle task which automates publishing Javadoc. d9cdabe

Actually I set the task to execute after bintrayUpload. But I'm not sure it is fit to you.

Why

See #40 .
opened by hotchemi 2
Publishing javadoc

Thanks for your great library guys.

I think publishing javadoc makes the library more understandable. Actually square's moshi just do it with their gh-pages branch. How do you think about of it?

opened by hotchemi 2

PureeBufferedOutput not work with SamplingFilter

PureeBufferedOutput#insertSync may be like below?

    @Override
    public void insertSync(String type, JsonObject jsonLog) {
        JsonObject filteredLog = applyFilters(jsonLog);
        if(filteredLog == null) {
            return;
        }
        storage.insert(type, filteredLog);
    }

opened by yoshiso 2

add PureeConfiguration#register() method

#register() is an alternative interface to #source().to(). What do you think of it?

I think #source() is confusing because it does not return PureeConfiguration.Builder even if it seems a builder pattern. In fact, I can't understand how it works in the first impression.

register(FooLog.class, new FooOutput().withFilter(...)), instead, shows it requires a pair of (1) a JsonConvertible log class and (2) a PureeOutput class which might have some filter classes.

Usage

From PureeConfigurationTest.java:

current

        PureeConfiguration conf = new PureeConfiguration.Builder(context)
                .source(FooLog.class).to(new OutFoo())
                .source(FooLog.class).filters(new FooFilter()).to(new OutFoo())
                .source(FooLog.class).filter(new FooFilter()).filter(new BarFilter()).to(new OutBar())
                .source(BarLog.class).filter(new FooFilter()).to(new OutFoo())
                .build();

new

        PureeConfiguration conf = new PureeConfiguration.Builder(context)
                .register(FooLog.class, new OutFoo())
                .register(FooLog.class, new OutFoo().withFilters(new FooFilter()))
                .register(FooLog.class, new OutBar().withFilters(new FooFilter(), new BarFilter()))
                .register(BarLog.class, new OutFoo().withFilters(new FooFilter()))
                .build();

Backward Compatibility

This PR keeps backward compatibility.

opened by gfx 2

Allow purging of old buffered logs

This PR adds a feature that allows purging of old logs stored in the buffer. This can be configured through OutputConfiguration#setPurgeAgeMillis()

To support this feature, PureeStorage needs to be modified to record log date and allow deletion based on log age. Since PureeStorage is an interface, modifying it directly will break existing clients that implements it so a new abstract class called EnhancedPureeStorage was introduced to add the new methods and support future changes.

The tests and demo project were also updated to support the new feature.

opened by epishie 1
Puree v5
Puree v5 I think...

Make interface same as Puree-Swift

Split sqlite tables between BufferedOutputs for fixing https://github.com/cookpad/puree-android/issues/61

(Optional?) Debugging feature

(Optional?) Support Kotlin by extension functions
opened by litmon 0
PureeBufferedOutput will never resume, if the flushSync function is skipped by locking storage with another PureeBufferedOutput instance
Hi.

Problem

When there are two or more PureeBufferedOutputs, if the flushSync function is skipped by locking storage with another PureeBufferedOutput instance, it will never resume.

Caused by

PureeBufferedOutput has RetryableTaskRunner and send log by the runner.

RetryableTaskRunner start by tryToStart function but the function skip if already 'future' exists. ★

PureeBufferedOutput call tryToStart function when receive new log. ★

PureeBufferedOutput#flushSync() will be skipped when 'storage' is locked. ★

RetryableTaskRunner resets 'future' only if it have done the send process (success / failed). ★

So, if there are multiple PureeBufferedOutputs and one locks the 'storage', the other instance's "future" will not be reset and the flushTask will not work. :dizzy_face:

Idea

I have no idea... :scream_cat:

thanks.
opened by sys1yagi 1

Releases(4.1.7)

4.1.7(Aug 7, 2020)

Remove Gson dependency from Puree and replaces the types used for representing the serialized JSON, JsonObject and the JsonArray, by String and List respectively
Source code(tar.gz)
Source code(zip)
v4.1.6(Jun 13, 2019)

Source code(tar.gz)
Source code(zip)
v4.1.2(Nov 1, 2018)

Source code(tar.gz)
Source code(zip)
v4.1.1(Sep 16, 2015)

Source code(tar.gz)
Source code(zip)

Owner

Cookpad

GitHub http://cookpad.github.io/puree-android/

Android Shared preference wrapper than encrypts the values of Shared Preferences. It's not bullet proof security but rather a quick win for incrementally making your android app more secure.

Secure-preferences - Deprecated Please use EncryptedSharedPreferences from androidx.security in preferenced to secure-preference. (There are no active