All

fuse on and off body sensing

as multimodal signals can provide complementary information for the same target

and thus achieve robust information inference

multiple occupants

large IoT systems that capture sensor data from more than one person at once -> requires association

user signal segment annotation

• use on body device for associating data with user (no manual effort as device already associated)

enhancing multimodal learning efficiency

indirect sensing leads to lack of direct comparable infomrmation

• indirect -> e.g. structural vibration

• raw data often not really interpretable and thus not easily compared for shared context

complementary leads to disassociation

• multiple modalities -> use complementary to gain overall more informaiton to achieve more efficient modeling

• due to being complementary -> captuer different type of information and thus not really share much information

mobility variance leads to spatiotemporal variation

• e.g. people moving around house -> association of on body sensor with off body sensor may vary (which one to associate) over time due to movement

determine wether two signal segments (from different senrors, (on body with n off body)) over the same period of time are associated (capture the same person)

association probabiliy

as long as sensors capture same physical activity -> there will be implicit shared context between two signal segments

assume that for structural virbation signals that are segmented as one activity (e.g. 5 seconds) -> there will be only a single wearable sensor associated

temporal convolutional neural network

CMA -> cross modal sensing signals segment level association scheme for multimodal IoT systems

AD-TCN -> framework for leanring segment level cross model representation and use learned model parameters to calculate amount of shared context between modalities

cross modal IoT device identification

IoT for occupant identification

shared 3D motion of human body parts captured by camera and IMU sensors for IoT device identifiaction

activity start/end time for fingerprinting co-located device pairing

=> have explicit context that is straightforward to see (e.g. rule based…)

=> TCN to discover limited association inforation that is a result form indirect sensing

use walking pattern / gait

voice

human body reflection

refraction, diffraction, absorption of radio signals

=> require labeled data to train

=> argument: difficult and impractical to asume availability of labeled data for each deployment…

=> use our approach to automatically label the data…

cross modal signal segment association scheme

association discovery temporal convolutional neural network

model requires input of same dimensionality

sensors have different sampling rate

synchronize timestamps

select lowest sampling rate of all available sensors

resample each other sensor inputs

by calculating least common multiple (kleinstes gemeinsames vielfaches)

then upsamle the higher sample rate signal to least common multiple

apply low pass

down sample ot lowest samping rate

sliding window over vibraiton signal and calcualte energy of each window

model out noise (gaussian)

apply threshold to energy

if above -> consider as event

activity swegmentation

interval-based lumping mehtod

look at consecutive event windows

and lump consecutive events that are together temporally shorter than a delta to one activity

lower and upper

if shorter than lower -> discard

if higher than upper -> divide and ggf. discard shorter

IMU (inertial measurement unit)

causal discovery between time series

method to infer causal relationship between pairs of multimodal sensing signals

train individual network for each vibration with all wearable sensors to estimate association relationship (-> train for associating 1 vibraiton with one of n wearables -> each virbration one wearable but wearable can have more…)

input: one vibration TS and n wearable TS (time aligned)

first association score layer (layer with a weight for each input) initiuaolized with same value

• updated during training with gradient descent

from association score layer, compute association score from output using softmax

branch out for actual training with TCN blocks that are used to predict the infrastructure signal (i.e. aim to predict the input as output including the wearable signals) and use this to trian the network

vibration is shifted left with 1 compared to prediction and wareable info

-> use vibration t-1 as baseline info and wearable t as prediciton basis to predict vibration t…

input time shifted vibraiton plus all wearable

go through weight layer with softmax

outputs go into individual TCN residual blocks (no detail)

pointwise convolution layer to integrate all outputs of blocks before

this output predicts the vibraion input + 1

use loss function to compare to actual input to train network

no not comparable between different networks

output is actually attention value and cannot represent association relationship directly

=> common represntaiton of association relationship needed

use association score to calculate divergence between vibration sensor and wearable sensors

and then use softmax to convert divergence to the association probability

idea: low association divergence -> low contribution to prediction

calculate by sqrt of euclidean norm of the output over the channels of each individual wearable sensor

then calculate softmax over all these wearable sensor specific values

yes multiple channels

e.g. accelerometer

gyroscope

floor vibraiton sensors

IMU (6-axis) sensors

two buildings with 6 humans and nine types of in home actiities of daily living

keyboard typing

using mouse

handwriting

cutting food

stir fry

wiping countertop

sweeping floor

vacuuming floor

open / close drawer

for each scenario fpour vibration sensors

10 times for 15 seconds

one building one human

same types of sensors

11 person

3 per house for data collection

three vibration sensors on surface of furniture

do their thing for around one hour simulatneously

use camera to verify later on (=> ground truth for evaluation)

ROC and AUC for activity recognition thresholding

• CMA and baseline methods

F1 score and accuracy for performance in selected (best) thresold

• intutitive overall eavluation on public and uncontrolled dataset

ture positive (right association wearable and vibration)

false positive (wrongly associatted)

regular signal similarity…

cosine similarity

surface similarity

max cross correlation

between IMU and vibration

two use cases implemented on public dataset (due to presence of activity and identity labels)

up to 31 percent better F1 than baseline

separate evaluation of direct (e.g. cut with knive -> cut creates vibration) and indirect ( vacuum floor -> motor creates vibraiton)

show ROC

was robust

choise three pairs

unassociated 0, 1, 2 (=> have a vibration without fitting IMU)

+> the more unassociated the lower the performacne

expl. multiple IMU equally not associated -> not efficient for distinguishing

increase number of weareable devices to alrger than 3

up to six

slight average decvrease in AUC

but still good

same as controlled but only for general performance (ROC, AUC, F1, Accuracy)

slightly better than in controlled dataset

occupant: improvement over random association but worse than ideal association

same in activity recognition