CRAWDAD mit/reality
收藏资源简介：
The authors have captured communication, proximity, location, and activity information from 100 subjects at MIT over the course of the 2004-2005 academic year. This data represents over 350,000 hours (~40 years) of continuous data on human behavior. Such rich data on complex social systems have implications for a variety of fields.last modified :2006-11-09release date :2005-07-01date/time of measurement start :2004-07-26date/time of measurement end :2005-05-05collection environment :Our study consists of one hundred Nokia 6600 smart phones pre-installed with several pieces of software we have developed as well as a version of the Context application from the University of Helsinki. Seventy-five users are either students or faculty in the MIT Media Laboratory, while the remaining twenty-five are incoming students at the MIT Sloan business school adjacent to the laboratory. Of the seventy-five users at the lab, twenty are incoming masters students and five are incoming MIT freshman.network configuration :We exploit the fact that modern phones use both a short-range RF network (e.g., Bluetooth) and a long-range RF network (e.g., GSM), and that the two networks can augment each other for location and activity inference. We logged cell tower ID to determine approximate location and at the same time we logged Bluetooth devices. Bluetooth is a wireless protocol in the 2.40-2.48 GHz range, developed by Ericsson in 1994 and released in 1998 as a serial-cable replacement to connect different devices.data collection methodology :The information we are collecting includes call logs, Bluetooth devices in proximity, cell tower IDs, application usage, and phone status (such as charging and idle), which comes primarily from the Context application. The study will generate data collected by one hundred human subjects over the course of nine months and represent approximately 500,000 hours of data on users' location, communication and device usage behavior.Tracesetmit/reality/blueawareTraceset of communication, proximity, location, and activity information.description: The authors have captured communication, proximity, location, and activity information from 100 subjects at MIT over the course of the 2004-2005 academic year. This data represents over 350,000 hours (~40 years) of continuous data on human behavior.measurement purpose: Social Network Analysis, Human Behavior Modelingmethodology: Every Bluetooth device is capable of device-discovery, which allows them to collect information on other Bluetooth devices within 5-10 meters. This information includes the Bluetooth MAC address (BTID), device name, and device type. The BTID is a hex number unique to the particular device. The device name can be set at the user's discretion; e.g., Tony's Nokia. Finally, the device type is a set of three integers that correspond to the device discovered; e.g., Nokia mobile phone, or IBM laptop. To log BTIDs we designed a software application, BlueAware, that runs passively in the background on MIDP2-enabled mobile phones. Bluetooth was primarily designed to enable wireless headsets or laptops to connect to phones, but as a byproduct, devices are becoming aware of other Bluetooth devices carried by people nearby. Our application records and timestamps the BTIDs encountered in a proximity log and makes them available to other applications. BlueAware is automatically run in the background when the phone is turned on, making it essentially invisible to the user. Bluedar was developed to be placed in a social setting and continuously scan for visible devices, wirelessly transmitting detected BTIDs to a server over an 802.11b network. The heart of the device is a Bluetooth beacon designed by Mat Laibowitz incorporating a class 2 Bluetooth chipset that can be controlled by an XPort web server. We integrated this beacon with an 802.11b wireless bridge and packaged them in an unobtrusive box. An application was written to continuously telnet into multiple BlueDar systems, repeatedly scan for Bluetooth devices, and transmit the discovered proximate BTIDs to our server. Because the Bluetooth chipset is a class 2 device, it is able to detect any visible Bluetooth device within a working range of up to twenty-five meters.last modified: 2006-10-17dataname: mit/reality/blueawareversion: 20050701change: the initial versionrelease date: 2005-06-01date/time of measurement start: 2004-07-26date/time of measurement end: 2005-05-05limitation: 1. Continually scanning and logging BTIDs can expend an older mobile phone battery in about 18 hours. While continuous scans provide a rich depiction of a user's dynamic environment, most individuals expect phones to have standby times exceeding 48 hours. Therefore BlueAware was modified to only scan the environment once every five minutes, providing at least 36 hours of standby time. 2. While the custom logging application on the phone crashes occasionally (approximately once every week), these crashes fortunately do not result in significant data loss. An additional small application was written to start on boot and continually review the running processes on the phone, verifying that our logging application is always running. Should there be a time where this is not the case, the application is immediately restarted. This functionality also ensures that logging begins immediately once the phone is turned on. However, while this logging application is now fairly robust and can be assumed to be running anytime the phone is on, the dataset generated is certainly not without noise. 3. By scanning only periodically every five minutes, shorter proximity events may be missed.hole: 1. All the data from a phone are stored on a flash memory card, which has a finite number of read-write cycles. Initial versions of our application wrote over the same cells of the memory card. This led to failure of a new card after about a month of data collection, resulting in the complete loss of data. When the application was changed to store the incremental logs in RAM and subsequently write each complete log to the flash memory, our data corruption issues virtually vanished. However, ten cards were lost before this problem was identified, destroying portions of the data collected during the months of September and October for six Sloan students and four Media Lab students. 2. Another source of missing data is due to powered-off devices. On average we have logs accounting for approximately 85.3% of the time since the phones have been deployed. Less than 5% of this is due to data corruption, while the majority of the missing 14.7% is due to almost one fifth of the subjects turning off their phones at night. 3. There is a small probability (between 1-3% depending on the phone) that a proximate, visible device will not be discovered during a scan. Typically this is due to either a low level Symbian crash of an application called the "BTServer", or a lapse in the device discovery protocol. The BT server crashes and restarts approximately once every three days (at a 5 minute scanning interval) and accounts for a small fraction of the total error. However, to detect other subjects, we can leverage the redundancy implicit in the system. Because both of the subjects' phones are actually scanning, the probability of a simultaneous crash or device discovery error is less than 1 in 1000 scans.error: 1. The ten meter range of Bluetooth along with the fact that it can penetrate some types of walls, means that people not physically proximate may incorrectly be logged as such. 2. An error comes from the phone being either explicitly turned off by the user or exhausting the batteries. According to our collected survey data, users report exhausting the batteries approximately 2.5 times each month. One fifth of our subjects manually turn the phone off on a regular basis during specific contexts such as classes, movies, and (most frequently) when sleeping. Immediately before the phone powers down, the event is timestamped and the most recent log is closed. A new log is created when the phone is restarted and again a timestamp is associated with the event. 3. A more critical source of error occurs when the phone is left on, but not carried by the user. From surveys, we have found that 30% of our subjects claim to never forget their phones, while 40% report forgetting it about once each month, and the remaining 30% state that they forget the phone approximately once each week. Identifying the times where the phone is on, but left at home or in the office presents a significant challenge when working with the dataset. To grapple with the problem, we have created a 'forgotten phone' classifier. Features included staying in the same location for an extended period of time, charging, and remaining idle through missed phone calls, text messages and alarms. When applied to a subsection of the dataset which had corresponding diary text labels, the classifier was able to identify the day where the phone was forgotten, but also mislabeled a day when the user stayed home sick. By ignoring both days, we risk throwing out data on outlying days, but have greater certainty that the phone is actually with the user. A significantly harder problem is to determine whether the user has temporarily moved beyond ten meters of his or her office without taking the phone. Empirically, this appears to happen with many subjects on a regular basis and there doesn't seem to be enough unique features of the event to accurately classify it. However, this phenomenon does not diminish the extremely strong correlation between detected proximity and self-report interactions. Lastly, while frequency of proximity within the workplace can be useful, the most salient data comes from detecting a proximity event outside MIT, where temporarily forgetting the phone is less likely to repeatedly occur.note: In return for the use of the Nokia 6600 phones, students have been asked to fill out web-based surveys regarding their social activities and the people they interact with throughout the day. Comparison of the logs with survey data has given us insight into our dataset's ability to accurately map social network dynamics. Through surveys of approximately forty senior students, we have validated that the reported frequency of (self-report) interaction is strongly correlated with the number of logged BTIDs (R=.78, p=.003), and that the dyadic self-report data has a similar correlation with the dyadic proximity data (R=.74, p~=.0001). Additionally, a subset of subjects kept detailed activity diaries over several months. Comparisons revealed no systematic errors with respect to proximity and location, except for omissions due to the phone being turned off.mit/reality/blueaware Tracesactivityscpan: Activity span logs.configuration: activity span logsformat: oid, endtime, starttime, person_oiddescription: Activity span logs.last modified: 2006-10-17dataname: mit/reality/blueaware/activityscpanversion: 20050701change: The initial versionrelease date: 2005-07-01date/time of measurement start: 2004-07-26date/time of measurement end: 2005-05-05callspan: Call span logs.configuration: call span logsformat: oid, endtime, starttime, person_oid, phonenumber_oid, callid, contact, description, direction, duration, number, status, remote "person_oid" refers to the person running the software on their phone, for which this call was logged. It is who this callspan is 'attached' to, and will always be attached to some person_oid. "direction" refers to the direction of the call from the perspective of this particular person/cellphone that recorded this callspan (the same as the person referred to by person_oid). Can be Incoming, Missed Call, or Outgoing. "phonenumber_oid" refers to the number 'on the other end' of the network, which may be a landline, a cell phone line, or even that phone network's voicemail. So in other words, person_oid and phonenumber_oid represent the two ends of the phone call, with the direction of the phone call represented in the direction field. If you want to utilize all 897921 callspan records, you might want to define these "calls" as between two phonenumbers, instead of as between two persons. So the call would exist between callspan.person_oid's phonenumber_oid, and the callspan.phonenumber_oid. In addition, if the callspan records a call between two people that were running the software and part of the study (they both are part of the study), then there are a few additional properties that will hold about the callspan: For some person src: src.oid = callspan.person_oid (for all calls) For some person dst: dst.phonenumber_oid = callspan.phonenumber_oid (only for in-network calls) There should also be a symmetric callspan going in the other direction. For some callspan Y: Y.person_oid == dst.oid Y.phonenumber_oid = src.phonenumber_oiddescription: Call span logs.last modified: 2006-10-17dataname: mit/reality/blueaware/callspanversion: 20050701change: The initial versionrelease date: 2005-07-01date/time of measurement start: 2004-08-03date/time of measurement end: 2004-12-25cellspan: Cell span logs.configuration: cell span logsformat: oid, endtime, starttime, person_oid, celltower_oiddescription: Cell span logs.last modified: 2006-10-17dataname: mit/reality/blueaware/cellspanversion: 20050701change: The initial versionrelease date: 2005-07-01date/time of measurement start: 2004-07-26date/time of measurement end: 2005-05-05coverspan: Cover span logs.configuration: cover span logsformat: oid, endtime, starttime, person_oiddescription: Cover span logs.last modified: 2006-10-17dataname: mit/reality/blueaware/coverspanversion: 20050701change: The initial versionrelease date: 2005-07-01date/time of measurement start: 2004-07-27date/time of measurement end: 2005-05-05devicespan: Device span logs.configuration: device span logsformat: oid, endtime, starttime, person_oid, device_oiddescription: Device span logs.last modified: 2006-10-17dataname: mit/reality/blueaware/devicespanversion: 20050701change: The initial versionrelease date: 2005-07-01date/time of measurement start: 2004-07-26date/time of measurement end: 2005-05-05
## 主数据集总述 研究人员于2004-2005学年期间，从麻省理工学院（MIT）的100名受试者处采集了通信、邻近度、位置与活动信息。该数据集包含超过35万小时（约40年）的人类行为连续观测数据。这类针对复杂社会系统的丰富数据集可对多个研究领域提供参考。 最后修改时间：2006-11-09；发布日期：2005-07-01；测量开始时间：2004-07-26；测量结束时间：2005-05-05。 ### 采集环境 本研究共使用100台预装自研软件及赫尔辛基大学Context应用程序版本的诺基亚6600（Nokia 6600）智能手机。其中75名受试者为麻省理工学院媒体实验室（MIT Media Lab）的学生或教职员工，剩余25名为毗邻该实验室的麻省理工学院斯隆商学院（MIT Sloan School of Business）新生。在实验室的75名受试者中，20人为入学硕士生，5人为麻省理工学院本科新生。 ### 网络配置 现代智能手机同时支持短距射频网络（如蓝牙（Bluetooth））与长距射频网络（如GSM），二者可互为补充以实现位置与活动推断。本研究通过记录基站ID获取受试者的近似位置，同时记录蓝牙设备信息。蓝牙是一种工作于2.40-2.48 GHz频段的无线通信协议，由爱立信（Ericsson）于1994年研发，1998年正式发布，用于替代串行线缆实现多设备互联。 ### 数据采集方法 本次采集的信息包括通话日志、邻近蓝牙设备、基站ID、应用使用情况及手机状态（如充电、空闲状态），数据主要通过Context应用程序获取。本研究由100名受试者历时9个月完成数据采集，累计产生约50万小时的用户位置、通信与设备使用行为数据。 ### 追踪集：mit/reality/blueaware **追踪集说明**：通信、邻近度、位置与活动信息追踪集。 **详细描述**：研究人员于2004-2005学年期间，从麻省理工学院的100名受试者处采集了通信、邻近度、位置与活动信息。该数据集包含超过35万小时（约40年）的人类行为连续观测数据。 **测量目的**：社交网络分析（Social Network Analysis）、人类行为建模（Human Behavior Modeling）。 ### 详细采集方案 所有蓝牙设备均支持设备发现功能，可采集5-10米范围内其他蓝牙设备的信息，包括蓝牙MAC地址（BTID）、设备名称与设备类型。BTID为对应设备的唯一十六进制编号；设备名称可由用户自定义，例如“Tony's Nokia”；设备类型为一组对应被发现设备的三个整数，例如“诺基亚手机”或“IBM笔记本电脑”。 为记录BTID，我们开发了BlueAware软件应用，该应用可在支持MIDP2的移动设备后台静默运行。蓝牙协议最初旨在实现无线耳机或笔记本电脑与手机的连接，但其附带功能可使设备感知附近用户携带的其他蓝牙设备。本应用会记录扫描到的BTID并添加时间戳至邻近度日志，同时将数据开放给其他应用。BlueAware会在手机开机时自动在后台运行，对用户基本无感知。 Bluedar设备部署于社交场景中，可持续扫描可见蓝牙设备，并通过802.11b无线网络将检测到的BTID无线传输至服务器。该设备的核心为Mat Laibowitz设计的蓝牙信标，搭载可通过XPort Web服务器控制的2类蓝牙芯片组。我们将该信标与802.11b无线网桥集成，并封装于小巧隐蔽的外壳中。另有一款应用程序可通过Telnet协议持续连接多台BlueDar系统，重复扫描蓝牙设备并将检测到的邻近BTID传输至服务器。由于采用2类蓝牙芯片组，该设备可检测最大工作范围内25米内的所有可见蓝牙设备。 最后修改时间：2006-10-17；数据集名称：mit/reality/blueaware；版本：20050701；变更记录：初始版本；发布日期：2005-06-01；测量开始时间：2004-07-26；测量结束时间：2005-05-05。 ### 数据集限制 1. 持续扫描并记录BTID会使老旧智能手机的电池在约18小时内耗尽。尽管连续扫描可完整刻画用户的动态环境，但多数用户期望手机待机时长超过48小时。因此我们对BlueAware进行了修改，将扫描频率调整为每5分钟一次，使手机待机时长至少可达36小时。 2. 手机端的自定义日志应用偶尔会出现崩溃（约每周一次），但幸运的是此类崩溃不会造成显著的数据丢失。我们额外开发了一款小型应用，可在手机开机时自动启动，持续监控手机运行进程，确保日志应用始终处于运行状态，若检测到应用异常则立即重启。该功能同时可确保手机开机后立即开始数据采集。尽管当前日志应用已具备较强的稳定性，可认为手机开机时日志程序始终运行，但生成的数据集仍存在一定噪声。 3. 由于仅每5分钟进行一次周期性扫描，时长较短的邻近事件可能会被遗漏。 ### 数据缺失情况 1. 手机数据均存储于闪存卡中，而闪存卡的读写循环次数有限。早期版本的应用会重复写入闪存卡的同一存储单元，导致新卡在约一个月的采集后即损坏，造成数据完全丢失。后续我们将应用修改为：先将增量日志存储于RAM中，待日志完整后再写入闪存，自此数据损坏问题基本解决。但在该问题被发现前，已有10张闪存卡损坏，导致6名斯隆商学院学生与4名媒体实验室学生在9月和10月采集的部分数据丢失。 2. 另一项数据缺失来源为设备关机。自手机部署以来，平均约有85.3%的时间存在有效日志，其中少于5%的缺失源于数据损坏，剩余14.7%的主要原因是近五分之一的受试者会在夜间手动关闭手机。 3. 扫描过程中存在1%-3%（依手机型号而定）的概率无法检测到邻近可见设备，通常源于名为“BTServer”的应用出现底层Symbian系统崩溃，或是设备发现协议失效。BTServer约每3天崩溃并重启一次（扫描间隔为5分钟），仅占总误差的一小部分。但我们可利用系统内置的冗余性弥补该问题：由于两名受试者的手机均会主动扫描，单次扫描中双方同时出现崩溃或设备发现错误的概率低于千分之一。 ### 误差来源 1. 蓝牙的10米通信范围及部分墙体穿透特性，可能导致非物理邻近的人员被错误记录为邻近状态。 2. 另一类误差源于用户手动关机或电池耗尽。根据收集的调研数据，受试者平均每月会遇到约2.5次电池耗尽的情况，五分之一的受试者会在特定场景下定期手动关机，例如上课、观影，最常见的场景为睡眠时。手机关机前会记录时间戳并关闭最新的日志文件，重启手机后会创建新的日志文件并同样记录时间戳。 3. 更严重的误差来源为：手机处于开机状态，但未被用户随身携带。根据调研，30%的受试者声称从未遗忘手机，40%的受试者表示每月约遗忘一次，剩余30%的受试者称每周约遗忘一次。在使用该数据集时，识别手机开机但被遗留在家中或办公室的时间段是一项重大挑战。为解决该问题，我们开发了“遗忘手机”分类器，其特征包括：长时间停留在同一位置、处于充电状态、未接来电、短信与闹钟均处于空闲状态。将该分类器应用于带有日记标签的子数据集时，可准确识别出手机被遗忘的日期，但也会将用户因病居家的日期误判为遗忘手机。若直接忽略这两类日期，虽可提高判断的准确性，但会丢失异常日期的有效数据。更棘手的问题是判断用户是否在未携带手机的情况下暂时离开办公室超过10米范围。经验表明，多名受试者会定期出现此类情况，且该事件缺乏足够独特的特征以实现准确分类。但该现象并未削弱检测到的邻近度与自我报告的交互行为之间的强相关性。此外，尽管工作场所的邻近频率具备一定研究价值，但最有价值的数据来自麻省理工学院以外的邻近事件，此类场景中临时遗忘手机的概率更低。 ### 验证说明 作为使用诺基亚6600手机的回报，受试者需填写基于网页的调查问卷，内容涵盖其当日社交活动及互动对象。将日志数据与调查问卷结果进行对比，可帮助我们评估数据集准确映射社交网络动态的能力。通过对约40名高年级学生的调研，我们验证了：自我报告的交互频率与记录的BTID数量呈强相关（R=0.78，p=0.003）；二元自我报告数据与二元邻近度数据也存在类似相关性（R=0.74，p≈0.0001）。此外，部分受试者在数月内记录了详细的活动日记，对比结果显示，除因手机关机导致的遗漏外，邻近度与位置数据不存在系统性误差。 --- ## 子数据集详情 ### 活动跨度日志（activityscpan） 配置：活动跨度日志 格式：oid, endtime, starttime, person_oid 描述：活动跨度日志 最后修改时间：2006-10-17 数据集名称：mit/reality/blueaware/activityscpan 版本：20050701 变更记录：初始版本 发布日期：2005-07-01 测量开始时间：2004-07-26 测量结束时间：2005-05-05 ### 呼叫跨度日志（callspan） 配置：呼叫跨度日志 格式：oid, endtime, starttime, person_oid, phonenumber_oid, callid, contact, description, direction, duration, number, status, remote 描述：呼叫跨度日志 字段说明： - `person_oid`：指运行本软件并产生该呼叫日志的受试者，即该呼叫跨度所属的受试者，必然对应唯一的受试者编号。 - `direction`：从记录该呼叫跨度的受试者（即`person_oid`对应的用户）视角出发的呼叫方向，可取值为：来电、未接来电、去电。 - `phonenumber_oid`：指通信网络另一端的号码，可为固定电话、手机号码，甚至运营商的语音信箱。简言之，`person_oid`与`phonenumber_oid`分别代表通话的两端，呼叫方向由`direction`字段体现。若需使用全部897921条呼叫跨度记录，建议将通话定义为两个电话号码之间的通信，而非两个受试者之间，即通话双方为`callspan.person_oid`对应的手机号码与`callspan.phonenumber_oid`。 - 若通话双方均为参与本研究且安装了本软件的受试者，则该呼叫跨度将满足以下额外属性： 1. 存在某受试者`src`，满足`src.oid = callspan.person_oid`（适用于所有呼叫） 2. 存在某受试者`dst`，满足`dst.phonenumber_oid = callspan.phonenumber_oid`（仅适用于研究内部通话） 3. 存在一条方向相反的对称呼叫跨度记录：存在某呼叫跨度`Y`，满足`Y.person_oid == dst.oid`且`Y.phonenumber_oid = src.phonenumber_oid` 最后修改时间：2006-10-17 数据集名称：mit/reality/blueaware/callspan 版本：20050701 变更记录：初始版本 发布日期：2005-07-01 测量开始时间：2004-08-03 测量结束时间：2004-12-25 ### 基站跨度日志（cellspan） 配置：基站跨度日志 格式：oid, endtime, starttime, person_oid, celltower_oid 描述：基站跨度日志 最后修改时间：2006-10-17 数据集名称：mit/reality/blueaware/cellspan 版本：20050701 变更记录：初始版本 发布日期：2005-07-01 测量开始时间：2004-07-26 测量结束时间：2005-05-05 ### 覆盖跨度日志（coverspan） 配置：覆盖跨度日志 格式：oid, endtime, starttime, person_oid 描述：覆盖跨度日志 最后修改时间：2006-10-17 数据集名称：mit/reality/blueaware/coverspan 版本：20050701 变更记录：初始版本 发布日期：2005-07-01 测量开始时间：2004-07-27 测量结束时间：2005-05-05 ### 设备跨度日志（devicespan） 配置：设备跨度日志 格式：oid, endtime, starttime, person_oid, device_oid 描述：设备跨度日志 最后修改时间：2006-10-17 数据集名称：mit/reality/blueaware/devicespan 版本：20050701 变更记录：初始版本 发布日期：2005-07-01 测量开始时间：2004-07-26 测量结束时间：2005-05-05
