I don’t write here anymore. You can now find me at coderchrismills.com.
A few months back someone asked me show them how to play audio from a users music library on iOS. I took this opportunity to demo music analysis program that covers a few things in one project.
- How to use blocks and NSOperationQueue to load data
- How to use MPMediaPickerController
- Simple audio analysis with vDSP
I was really hoping to write a more tutorial style post on how to do all this, but sadly I'm not going to have the time. That being said there's a few code snippets that I think will help people a lot if they're doing something similar. The code here can be a bit touch to read with the way the formatting is done, but it’s all available on GitHub.
First up, loading music using the MPMediaPickerController. In the app I’m using a button open picker modal
- (IBAction)showMediaPicker:(id)sender
{
MPMediaPickerController *mediaPicker = [[MPMediaPickerController alloc] initWithMediaTypes: MPMediaTypeAny];
mediaPicker.delegate = self;
mediaPicker.allowsPickingMultipleItems = NO;
mediaPicker.prompt = @"Select song to analyze";
[self presentModalViewController:mediaPicker animated:YES];
mediaPicker = nil;
}
So now we need to respond to the user selecting a song. To do this we need to implement
- (void) mediaPicker: (MPMediaPickerController *) mediaPicker didPickMediaItems: (MPMediaItemCollection *) mediaItemCollection
We’ll also need to define an MPMusicPlayerController in our .h file since we’re going to also be handling pausing and playing of the selected track.
MPMusicPlayerController *musicPlayer; // In the header file
- (void) mediaPicker: (MPMediaPickerController *) mediaPicker didPickMediaItems: (MPMediaItemCollection *) mediaItemCollection
{
if (mediaItemCollection) {
NSArray *songs = [mediaItemCollection items];
[musicPlayer setQueueWithItemCollection: mediaItemCollection];
[loadingView setHidden:NO];
[activityIndicator setHidden:NO];
[activityIndicator startAnimating];
NSOperationQueue *queue = [[NSOperationQueue alloc] init];
[queue addOperationWithBlock:^{
[self loadSongData:(MPMediaItem *)[songs objectAtIndex:0]];
[[NSOperationQueue mainQueue] addOperationWithBlock:^{
[activityIndicator stopAnimating];
[activityIndicator setHidden:YES];
[loadingView setHidden:YES];
// Start the timer to sample our audio data.
float interval_rate = ((songLength / songSampleRate) / MAX_FREQUENCY);
[self setAnalysisTimer:[NSTimer scheduledTimerWithTimeInterval:interval_rate
target:self
selector:@selector(doAnalysis:)
userInfo:nil
repeats:YES]];
[musicPlayer play];
}];
}];
}
[self dismissModalViewControllerAnimated: YES];
}
In the above code you’ll notice a function called loadSongData, this is the meat of loading the song data. I’ve removed a bit of the full function which you can find on the GitHub page.
- (void)loadSongData:(MPMediaItem *)mediaitem
{
NSURL* url = [mediaitem valueForProperty:MPMediaItemPropertyAssetURL];
AVURLAsset * asset = [[AVURLAsset alloc] initWithURL:url options:nil];
NSError * error = nil;
AVAssetReader * reader = [[AVAssetReader alloc] initWithAsset:asset error:&error];
AVAssetTrack * songTrack = [asset.tracks objectAtIndex:0];
NSArray* trackDescriptions = songTrack.formatDescriptions;
numChannels = 2;
for(unsigned int i = 0; i < [trackDescriptions count]; ++i) {
CMAudioFormatDescriptionRef item = (__bridge CMAudioFormatDescriptionRef)[trackDescriptions objectAtIndex:i];
const AudioStreamBasicDescription* bobTheDesc = CMAudioFormatDescriptionGetStreamBasicDescription (item);
if(bobTheDesc && bobTheDesc->mChannelsPerFrame == 1) {
numChannels = 1;
}
}
const AudioFormatListItem * afli = CMAudioFormatDescriptionGetRichestDecodableFormat((__bridge CMAudioFormatDescriptionRef)[trackDescriptions objectAtIndex:0]);
songSampleRate = afli->mASBD.mSampleRate;
DebugLog(@"%f", afli->mASBD.mSampleRate);
NSDictionary* outputSettingsDict = [[NSDictionary alloc] initWithObjectsAndKeys:
[NSNumber numberWithInt:kAudioFormatLinearPCM],AVFormatIDKey,
[NSNumber numberWithInt:16],AVLinearPCMBitDepthKey,
[NSNumber numberWithBool:NO],AVLinearPCMIsBigEndianKey,
[NSNumber numberWithBool:NO],AVLinearPCMIsFloatKey,
[NSNumber numberWithBool:NO],AVLinearPCMIsNonInterleaved,
nil];
AVAssetReaderTrackOutput * output = [[AVAssetReaderTrackOutput alloc] initWithTrack:songTrack outputSettings:outputSettingsDict];
[reader addOutput:output];
output = nil;
NSMutableData * fullSongData = [[NSMutableData alloc] init];
[reader startReading];
while (reader.status == AVAssetReaderStatusReading){
AVAssetReaderTrackOutput * trackOutput = (AVAssetReaderTrackOutput *)[reader.outputs objectAtIndex:0];
CMSampleBufferRef sampleBufferRef = [trackOutput copyNextSampleBuffer];
if (sampleBufferRef){
CMBlockBufferRef blockBufferRef = CMSampleBufferGetDataBuffer(sampleBufferRef);
size_t length = CMBlockBufferGetDataLength(blockBufferRef);
UInt8 buffer[length];
CMBlockBufferCopyDataBytes(blockBufferRef, 0, length, buffer);
NSData * data = [[NSData alloc] initWithBytes:buffer length:length];
[fullSongData appendData:data];
CMSampleBufferInvalidate(sampleBufferRef);
CFRelease(sampleBufferRef);
data = nil;
}
}
[self setSongData:nil];
if (reader.status == AVAssetReaderStatusFailed || reader.status == AVAssetReaderStatusUnknown){
DebugLog(@"Something went wrong...");
return;
}
if (reader.status == AVAssetReaderStatusCompleted){
[self setSongData:[NSData dataWithData:fullSongData]];
}
fullSongData = nil;
reader = nil;
asset = nil;
outputSettingsDict = nil;
songLength = [[self songData] length] / sizeof(SInt16);
songWaveform = (SInt16 *)[[self songData] bytes];
DebugLog(@"Data length %lu", songLength);
UIImage *waveimage = [self audioImageGraphFromRawData:songWaveform withSampleSize:songLength];
if(waveimage != nil)
{
[fullWaveformImageViewLandscape setImage:waveimage];
[fullWaveformImageViewPortrait setImage:waveimage];
}
[self setFullWaveformImage:waveimage];
nLastPlayIndex = 0;
}
Alright, now the song data is ready to use and we can play the song with [musicPlayer play]. So far we’ve gone over using blocks to load a song without locking up the UI and selecting songs from your music library. Only thing left to quickly go over is the vDSP library. It’s actually the simplest thing you can imagine as the Accelerate framework in my opinion is very easy to use. Here’s the FFT function I’m using in the app,
- (void)computeFFT
{
// Only works iOS 4 and above
for (NSUInteger i = 0; i < MAX_FREQUENCY; i++)
{
input.realp[i] = (double)fftWaveform[i];
input.imagp[i] = 0.0f;
}
/* 1D in-place complex FFT */
vDSP_fft_zipD(fft_weights, &input, 1, 6, FFT_FORWARD);
input.realp[0] = 0.0;
input.imagp[0] = 0.0;
}
Before you can use this function you need to create the weights, in my case I'm doing this in the viewDidLoad function.
fft_weights = vDSP_create_fftsetupD(6, kFFTRadix2); // 6 is the base 2 exponent.
Well, that's about it. There's a lot more in the code sample and I hope you check it out. The last thing I'd like to mention is that this app also draws both the waveform and spectrum of the audio being played. The majority of the code that draws these is from this StackOverflow post which also goes over a lot of what I've written here.
Chris Mills 